Questions and Answers from TED reusers Workshop
14 June 2023
These are the consolidated Questions and Answers (Q&A) addressed during the TED re-users workshop of 16 June 2023. Replies to questions posted live during the workshop were reviewed and regrouped or modified for a more cohesive and complete response.
Documentation
- Is there a centralised document that gives full definitions for the variables and options for the different search fields to aid understanding? In other words, are there definitions for what each of the legal basis mean, what each notice types mean, etc.
Answer: The eForms Regulations is the authoritative source of information about fields and definitions. Useful information is also available in the TED Developers Portal (https://docs.ted.europa.eu). - The Developers Documentation contains a warning about the field’s repository: “…However we currently do not guarantee the stability of the structure or content described here.” What does that mean about the usability of fields.json? https://docs.ted.europa.eu/eforms/latest/fields/index.html#field-repository.
Answer: We provide as much information as possible, as well as updates. We publish relevant information as fast as possible; however, there may be changes implemented before the documentation is available, thus the warning. - Is the fields.json file part of the SDK (Software Development Kit) or the documentation? Is it up to date?
Answer: The SDK is updated and contains the latest information available. Mistakes or bugs that are found are corrected in the upcoming releases. - It would be nice if you could create a short guide that summarizes in a concise manner what a re-user would have to do to replicate what is essentially done using the FTP server today, i.e., just fetching all notices daily, that could be available even after this workshop.
Answer: We understand the need of documentation so we will make it available if possible. In any case, the same packages (daily and monthly) that are available in the FTP server exist also in the “XML bulk downloads” section of the current TED for logged in users. In TED 2.0 the same “XML bulk downloads” section will exist and will be accessible by any user without the need to be logged in. - Is there a list detailing all significant structural changes between the current forms and eForms (e.g., in eForms the “deadline” field is available at “Lot” level)?
Answer: There is mapping between TED-XML standard forms and new eForms available on SIMAP at the following link. - Is there any way to help track tenders that could be categorised into different policy objectives such as the green and digital transition or various cohesion policies?
Answer: With eForms, it will be possible to better categorise procurement features. There will be fields for that and some of them will be mandatory. It should be stressed though that the Contracting Authorities (the Buyers) are responsible for the correctness, accuracy and completeness of the information that is published on TED.
FTP vs HTTPS - Will the FTP service continue to be supported?
Answer: No, the FTP will not be supported. It will be replaced by HTTPS. See also the answer to question 4 above. - How can the current usage of the FTP server be replicated - i.e., just fetching all notices daily. For many re-users, the primary use case is fetching ALL the data. Search features, etc. would then be provided in their own services, so how to do queries directly in TED etc. is not really of primary interest.
Answer: The process is described in the presentation given during the workshop of 14 June 2023 available at https://op.europa.eu/documents/d/ted-reusers-workshops/spotlight-on-reuse-1. See also the answer to question 4 above. - When will the current FTP be decommissioned?
Answer: The current FTP will be decommissioned when the new TED website goes live. - Is there a specific date/timeframe for when those of us using FTP (from the survey it was 50% of re-users) to do the transition to HTTPS? And when will the final HTTPS service be available?
Answer: We will do our best to ensure a proper transition period making FTP available for the new TED website. See also the answer to question 4 above. - Is there a plan to deprecate any of the current ways to access TED (open access to the XMLs via FTP)? Answer: With new TED the current access will be discontinued. There will be no access through FTP, rather via HTTPS. You will simply have to replace one URL with another URL. See also the answer to question 4 above.
API - What is the URL to the new API?
Answer: We will provide the URL when the system will be deployed in the Production environment. - Will the current URL https://ted.europa.eu/api/v3.0/notices/search used for searching API going to change?
Answer: Yes, the URL will change. It will refer to the new TED website. We will have another URL and we will provide it to the re-users.
DATA RETRIEVAL - Is there any machine-readable endpoint to get the current OJ S number in the new TED website?
Answer: There is no machine-readable endpoint to get the current OJ S number, however the information linking the publication number and the date is available in the release calendar that is published in TED. - For the HTTPS download URL we need the current OJs-ID (https://ted.europa.eu/xml-packages/daily-packages/2023/06/20230601_2023104.tar.gz). How do we know what the current OJs ID is?
Answer: With the new TED, you will only need the OJ S number. It will be easier. For example in order to get the first OJS of the year 2023 i.e. the 001 you use the year and the OJS number padded with 2 zeros (00) e.g.: https://blablabla/orchestrator/packaging/daily/202300001 As stated in the answer to question 14 the OJ S publication numbers and the date of each publication can be found in the release calendar that is published on TED. - When do you expect the HTTPS download in a format like https://blablabla/orchestrator/packaging/daily/202300001 be ready?
Answer: The HTTP download will be available from the day that the new TED will be live. The structure of this package may be adjusted. - Is there a list of all the packages published in a month in the new HTTPS download, similar to the one available in (e.g.: https://ted.europa.eu/xml-packages/daily-packages/2023/06/ lists the packages for June 2023).
Answer: There will be a similar structure for monthly packages in the new TED. See also the answer to question 4 above. - Do RE-Users really use the search on the TED-website instead of downloading and processing the bulk data?
Answer: There are various possible ways of using data. It may happen that different re-users use different methods. - Are multiple searches also possible through the advanced search in the new TED? e.g., "Contract Award Notice" + "Lot has been awarded? NO".
Answer: Yes, multiple searches are possible. - Where can the codes for the expert search be found?
Answer: Codes for the expert search can be found via the user Interface of the expert search. In the current TED, there is a “help page”. It will also be available in the new TED. The expert search of the new TED is more intuitive than the one of the current TED. - Regarding historical data (in RDF or TED 2.0) - does the current limit of 10 years still apply? Or is there a chance to get access to an even greater scope of data (beyond 10 years back from today)?
Answer: There is currently a limit of 10 years but the possibility to go even further is not excluded.
TED SEMANTIC WEB SERVICE (TEDSWS) - When will the new semantic web service be available for re-users?
Answer: The TED Semantic WEB Service will be available by the end of the summer 2023. [N.B.: in place since 28/8/2023] - Is there a sample for how RDF will look like?
Answer: Samples are available at the following link: https://github.com/OP-TED/ted-rdf-mapping/blob/main/mappings/package_F03/output/form_number_F03_S05/001331-2022/001331-2022.ttl - For someone not familiar with a” Semantic Web Service” or RDF are there some examples of use cases for this, and of how they would compare to just getting raw TED XML files?
Answer: To get the data for one year of XML files or eForms, for example, you will need to parse XML files and use an application for doing it. If you read RDF files, you’ll be able to parse data much faster. The RDF schema does not change, and the semantics do not change either. You can go to the SPARQL endpoint that will be available in the future, send the query in SPARQL query language (SQL) and you will get the data that you asked for. It will take some exploration at the beginning, but it is not difficult. - Will RDF files replace XML files? Will we stop downloading XML files and start downloading RDF files from October on?
Answer: RDF is not going to replace the XML files. The current TED XML files will continue to exist even after October and will be downloadable and reusable, alongside eForms (which are also XML). RDF is an extra; the intention is that in the future, whatever file that is available in XML we will also be available in RDF. - Will old TED data be eventually available in the new TED semantic web service? What is the rollout plan? Will the RDF be available before total discontinuation of the old TED schema?
Answer: The general idea is to gradually make TED data available in RDF. We will map some of the older schemas and then we will be able to convert older notices them in RDF. You may give us indications on what you would like to have and what would you need it for, for us to take your needs into consideration. - Would RDF allow to download daily-packages via code?
Answer: Yes, it will be possible. Once in Production, further details can be given. - How would RDF be available to the user? Can it be accessed programmatically in the TED semantic website?
Answer: Yes, the entire semantic web data can be used programmatically without any human intervention. It is not for consumption of people but for machines.
TEST ENVIRONMENT - Is there a development / acceptance environment of the new TED? Will there be a possibility to test the HTTPS package download before the FTP is turned off?
Answer: A test environment should be available mid-October. - In order to test our implementation with a proper fallback mechanism would it be possible to keep the current system running in parallel with the old system?
Answer: There will be a “parallel run” environment for test purposes to make sure that everything runs correctly. However, it is not possible to have two systems publicly accessible at the same time. - Can re-users have access to the Parallel Run environment?
Answer: Access to the Parallel Run environment is limited to the Publications Office, however a similar test environment will be made available.
CONTACT - Via which channel should we get in contact with you in order to describe our needs as data consumers?
Answer: The best way to reach out is through the TED helpdesk (https://ted.europa.eu/TED/misc/contact.do). In this way, every request is properly registered and will be followed. Please choose the most appropriate heading ("Reusing TED data") and provide us your suggestions. We would like to provide a more dedicated service through face-to-face meetings to discuss specific points, but this is a service we cannot guarantee right now with the level of quality that we would like to offer.
MISCELANEOUS - Why are the attendees of this workshop called "RE-users" and not simply “USERS? Who's the original user of the notice data, then? Are we not primary consumers of the data and APIs?
Answer: There are various types of users of TED, with various needs, and we hope we address all of them. We distinguish between those who come to TED and use the user interface to extract information for immediate consumption (“users”), and those – like you – who fetch data, process it, and provide it to other entities (“re-users”). - Who are the TED re-users?
Answer: Access to TED is free and anonymous so we do not have complete and accurate information about re-users. Every information you can give us is much welcome. That is also why we organised these events: to find out more; to collect your information and understand your needs. - How can notices published in the old schema, e.g., F02 published before the change to eForms, be changed? And how should the award notices, e.g., F03, be published?
Answer: After 25 October, only eForms notices can be published. To modify a notice previously published, or to publish a contract award notice, contracting authorities may use the conversion tool offered by the Publications Office. The Publications Office will not do any conversion.