Minutes of the Workshop on eForms for TED data reusers (27/09/2022)
The eForms Workshop of 27 September 2022 provided an overview of the impact of eForms on re-users of the TED website.
The recordings of the live session are available in the agenda section along with the PowerPoint slides of each presentation.
These minutes include replies to the questions addressed during the webinar. The questions and answers have been reviewed and may have been regrouped or modified for this written version to allow a more cohesive and complete overview.
LEGAL BASIS AND TIMELINE FOR EFORMS
eForms were adopted via the Commission Implementing Regulation (EU) 2019/1780 of 23 September 2019. More information about eForms is available at https://single-market-economy.ec.europa.eu/single-market/public-procurement/digital-procurement/eforms_en
The use of eForms is voluntary from 14 November 2022 and mandatory from 25 October 2023. An amendment to the Regulation is under preparation and should be adopted in November 2022.
Are you aware if there are actually any eSenders ready to send notices in eForms by 14 November?
We know that some are advancing with their developments (a minority). We will meet eSenders this week and will ask that; we are not aware of any of them going live with eForms on 14 November. Note that around 20% of notices come from buyers that don’t submit notices via eSenders; they may choose to start using eForms via eNotices2 as soon as it’s available, although we don’t expect many will do so.
Do you have a list of the countries that provide data for publication in TED from one single source (centralized)?
We do not have a list of countries providing data from one single source. However, the analysis of the statistical data available in SIMAP https://simap.ted.europa.eu/en_GB/web/simap/statistical-production-files can give an indication of the origin of the data per country.
CHANGES IN TED WEBSITE
The current TED website will be updated to be able to publish daily both current TED XML forms and eForms from 14 November 2022.
The current TED website will be replaced with a new TED website by the summer of 2023. The future TED website will publish daily both TED XML forms and eForms until 24 October 2023.
eForms become mandatory on 25 October 2023. From that date, the future TED website will only publish daily eForms.
Notices published in the current TED website and in the future TED website will continue to be available for search and reuse in the formats on which they were published.
Data format
Between 14 Nov 2022 and 25 Oct 2023 are you going to publish both eForms and TED XMLs, in parallel?
Yes.
Until October 2023, some notices will be in the TED XML format, some in the eForms format. Not all will be in TED XML. Is this correct?
Yes.
Will notices sent by an eSender in eForms format, also be available from TED in the old FTS format? Or will reusers be forced to handle both the new eForms format AND the old FTS concurrently?
Data may be in either one or the other format – according to the format sent by the buyer/eSender – but never in both formats. The Publications Office will make no conversion. Reusers will have to handle both eForms and the current schema at the same time.
Will a published tender have two formats available (old TED XML and eForms)?
Each notice or published tender will maintain the format in which it has been created/sent.
If we have to handle the new types concurrently, do we need be ready to parse eForms XML from November 2022?
Yes.
Would we miss notices if we would only request notices in the old TED XML schema?
Yes.
My understanding is that from 14th November 2022 to 25 October 2023, TED reusers will need to support both the old AND the new format to ensure they capture all new notices published by the buyers. Can you please confirm my understanding?
Yes, we confirm. During the transition period (November 2022 – October 2023) we will publish both formats – according to the original format of the notice – and you will be able to consume both types of notices. From 25 October 2023 only eForms will be processed. Old notices will be still there available to be retrieved.
Will the ability to render TED-XML notices into HTML/PDF keep running? It may be relevant to render these older notices for a long time.
Yes, the current notice viewer for the current TED-XML will continue to be available, as it is needed on TED to show notices published over the past 10 years (data available from 01/01/2014).
Are there any plans of offering a JSON alternative?
No.
Are you going to expose a SPARQL end point over the overall TED data in order to enable independent search on the knowledge base online?
A TED semantic web service that would open up data for everyone is planned for next year. Data will gradually be introduced during 2023 and 2024 to allow querying on different schema (current 2.0.9 schema and eForms).
Conversion
Are you converting old TED XML (before 2007) to eForms?
No, the Publications Office is not converting any notice. Each notice will be received and published in the format it has been sent in by the buyer/eSender. If sent as eForms, it will be published as eForms.
Content of daily packages
Will the daily-packages with XML files change as of a result of the new eForms? Or will it still be possible to retrieve information from XML format in the future?
The location of the daily-packages will not change. Reusers will be able to download daily packages via HTTP through the page https://ted.europa.eu/TED/misc/xmlPackagesDownload.do
Retrieval of daily deliveries: FTP and HTTP
Will there be any changes to the FTP delivery before 25 October 2023?
There will be changes due to the arrival of the new TED website which is scheduled for summer 2023.
In the current TED portal, reusers can download XML notices in bulk via HTTP or FTP.
In the future TED portal, reusers will be able to download XML notices in bulk via HTTP only.
Will the FTP bulk download of notices be stopped in a few weeks (when new TED Website will be released)?
Can you please clarify when the FTP bulk download will be stopped?
The FTP download will be stopped when the future TED website will go live in the summer of 2023.
On FTP there are XMLs. On HTTP there are formatted notices. So, if you stop providing FTP access, how do we retrieve the XML files?
It is possible to retrieve the XML file in bulk via HTTPS, even after the FTP will be disabled.
Today we are using the FTP server to get TED XML files in bulk daily. Since the FTP server will not be available after October 2023, can we get TED XML files in bulk using the API?
No. With the new TED, a new API will allow users to retrieve the metadata of the notice. Among the results, there is the URL of the format. But you will not be able to retrieve the content via API.
Today, in the current TED portal, reusers can download XML notices in bulk via HTTP or FTP.
In the future TED portal, reusers will be able to download XML notices in bulk via HTTP only.
In the daily bulk download via FTP are eForms and current TED-XML notices mixed in the same compressed download?
Yes.
As FTP will be replaced by HTTPS with the new version of the website, do you have an example using curl without manually logging in?
It is not necessary to log in.
API/Documentation
When will the new APIs documentation be available? It is still marked as “coming soon” on https://docs.ted.europa.eu/home/index.html
The preview environment listed on https://docs.ted.europa.eu/home/eforms/preview/index.html does not provide the URL for new search API (TBD) nor for new eForms viewer.
The API documentation was published in mid-October at https://docs.ted.europa.eu/api/index.html but please note that it addresses the new eForms APIs for the validation and visualisation of eForms notices.
The eForms TED Viewer 2022 is also now available in the Preview environment.
The documentation for the TED website API continues to be available in SIMAP at https://simap.ted.europa.eu/web/simap/developers-corner
When will the V3 of the TED website API be available?
The new v3 API for TED isn’t publicly available yet, it will be deployed directly in TED production and there is no Preview environment for TED and this API.
Is the new (V3) API able to handle a combination of old and new values, e.g., 2 and 3 digit CY's?
No. Only the 3-letter code is supported. You have to adapt your query.
How are we supposed to be ready for the new format if no documentation is available before going live?
The schemas and documentation have been made available through SIMAP since December 2019 and through the Software Development Kit (SDK).
The Regulation and the procurement Directives set the obligation for the Publications Office to receive, process and publish notices. We are doing all the best we can to help reusers as well – and will keep doing so.
Can we use the new eForms viewer API with an "old" TED-XML? and an eForms notice with viewer API v1?
No, there are two viewers that each handle only one schema. The new TED Viewer 2022 API only handles eForms XML. The existing TED Notice Viewer continues to only handle the TED XML.
SDK/Schema
It seems there are missing elements in the SDK schemas (.xsd) Error: Error generating classes for schema 'maindoc/UBL-ContractNotice-2_3'. - The element 'urn:oasis:names:specification:ubl:schema:xsd:CommonExtensionComponents-2:UBLExtensions' is missing
The schema is composed of a collection of files stored in two different folders:
- Maindoc (that contains the document types), and
- Common (that contains the common files)
Both folders must be downloaded in their entirety with their relative positions kept as the common files are imported using a relative path reference. Include mechanism is also used for extensions.
The performed action that generated the error message “Error: Error generating classes for schema ...” is not specified. It should be checked whether it supports inclusion and import operations. When it does and errors still occur, an option is to have the files explicitly listed on the command line.
Could you give an example of some these co-constraints?
Example 1
One date has to follow the other. Duration Start Date (BT-536) value must be earlier than Duration End Date (BT-537) value (BR-0043-0001, DG|108).
Example 2
One date shall not be later than 30 days after the other.
Example 3
If Procedure Legal Basis (BT-01) value is different from "Directive 2009/81/EC", then Unpublished Access Date (BT-198) value must be earlier than Notice Dispatch Date (BT-05) value + 5 years. (BR-0156-0001, DG|383).
Are the 500+ rules to improve data quality totally new rules or are they already in use?
Both. They group already existing rules as well as new rules (for deeper controls and for controls on new fields).
Will a list of these constraints be available somewhere? Or is that the XSD files?
The rules are expressed in Schematron. The Schematron files are part of the SDK.
Rules are excluded from the XSD. Any delay in information publication would make the XSD useless if rules were integrated into it.
The rules can also be browsed at https://docs.ted.europa.eu/eforms/1.2.1/reference/index.html
I understand that this is still a work in progress, but when I tested SDKs the example XMLs provided about a month ago, I got errors as they were missing elements that were supposed to be there. Will we soon be provided complete eForms for testing purposes, preferably on a regular basis?
We are currently improving version 1.0.0 of the SDK. Please do indicate any technical SDK errors you encounter on Github. Of course, Github does not replace the Helpdesk functions for questions not related to the SDK.
Examples and rules are released with every new SDK. As new rules are added, the file sample is corrected based on the validation reports. There is a known issue with our Schematron implementation for development. Schematrons will be modified in order to behave in the same way regardless of the implementation. For further details, users can visit Github https://github.com/OP-TED/eForms-SDK/discussions/75
The format of the publication number will change from 6 to 8 digits. Can you clarify the issue and its impact?
After 14/11/2022, the bulk FTP will have both formats. Is there a way to differentiate which notices are in each format prior to collecting? Is there a special naming convention? Or they will all be mixed together?
The TED website currently allows a maximum of 999 999 notices to be published per year (i.e. 6 digits, e.g. 1-2022, 654321-2022).
To enable more than one million notices to be published per year, we have increased the publication number to 8 digits (e.g. 1-2024, 87654321-2024).
Due to this extension, the XML packages will contain two formats of the XML files:
- for eForms notices the format will have 8 digits (e.g. 00654321_2022.xml)
- for TED schema notices the format XML file will have 6 digits (e.g. 123456_2022.xml).
The publication number will have 8 digits for eForms notices and 6 digits for TED schema notices.
When will the final version of the eForms SDK be released? (as for now the disclaimer on the Github states that the current version is still a non-definitive one).
The disclaimer is outdated and we need to change it. There will never be a “final” version, changes will be continuous. Version 1 is ready for Production, i.e., the structure of the SDK is stable but not the content (like the schema). When the need arises, we will consider a new version and inform you. Before 14/11/2022, we expect two minor releases of the SDK to add and refine elements like the schema (to add the new fields from the 2022 amendment to the eForms Regulation), rules, codelists, translations. This need of new versions is due to the new eForms Regulation that will be published soon.
What is the XML tag to get the status of the Notice (e.g., Open/Awarded, etc.)?
There is a dedicated Business Term that provides information about the award status of a lot (Winner Chosen BT-142), which will contain a value from the codelist Winner Selection Status.
The XML element that conveys that information is cbc:TenderResultCode.
Testing/Access to testing environments of the Publications Office
Will you provide sample eForms datasets for testing purposes?
Where can we download a set of eForms to test our processing on?
My understanding is that there will be no eForms available to test on before going live on 14 November. Is this correct?
Some examples of eForms XMLs are available for testing in the SDK at the following URL: https://github.com/OP-TED/eForms-SDK/tree/main/examples.
As soon as we have a bigger set of eForms to share with reusers, we will do it.
It is also possible to generate your own eForms for testing purposes using the eNotices2 application in the Preview environment (see https://docs.ted.europa.eu/home/eforms/preview/index.html).
eNotices2
We used TED eNotices2 to generate test eForms and it is not working anymore (like for a month or so now). Are there prospects to make it work again?
Yes. You can generate your own test notices. eNotices 2 is buggy at the moment, the next version will be available online in late October. At the moment we are testing it.
Miscellaneous questions and comments
Could you please explain the difference between light and standard procurement notice?
This refers to types of notices that have “lighter” publication requirements. For example, in Directive 2014/24/EU, see articles 74 and 75 about the award of contracts for social and other specific services.
When you make a search, for example, for the Spanish contract notices, you are allowed to download it in the XML format, but only the Contract notices that are in the first page. Is there any option to download all the notices, not only the first 25?
Downloading notices via the user interface has limits. The user can extend this limit to 100 items by changing the “Number of items per search result page” on the page Preferences.
We suggest to use https://ted.europa.eu/api/swagger-ui.html or the option of XML bulk download.
Please clarify how to filter the search by country or CPV, for example, in a bulk download.
Download all notices, then execute a query for each notice to get the info you want to retrieve.
Example
The user wants to download the notices from OJS 188/2022 where the country of the buyer is ‘Spain’.
The user executes the query ‘OJ=[188/2022] AND CY=[ES]’ via ‘TED website API’ to get the list of publication numbers.
The user downloads OJS 188/2022 using the option XML bulk download and retrieves the XML from the downloaded XML package.
It would be great to turn on directory listening at the bulk download folder to be able to get all archives without guessing the names of the archives.
We will assess the feasibility of this suggestion.
I think it would be much appreciated if a guide on how to get all data from TED daily is created and made available before 14 November.
We will try our best to provide as much documentation as possible in due time.
Would it be an idea to appoint an internal advocate for data reusers? Someone who could have as part of their role to understand the needs of this stakeholder group. It seems you are well sorted on internal representation of the eSenders.
We will assess the feasibility of this suggestion.
Can the language used to render the notice, be any official language of the EU no matter what language the notice is published in?
Yes. Free text (i.e., title of the notice, field II.1.1) is always in the original language. If you change the User Interface (UI) language, you will have the free text in the original language and the code values and labels in the UI language.
1) Are notices published only in their original language? Or it is possible to have multiple languages (as before)?
2) Does the full text of the notices need to be created and translated as before (based on the tags)?
The free text is in the original language of the notice and it is not translated (except from notices coming from EU institutions, agencies and bodies).
Answer to the 1st question:
On GitHub eForms-SDK the folder examples contains an example of notice with several languages - cn_24_multilingual.xml. The eForms notices can be published in several languages.
Answer to the 2nd question:
On docs.ted.europa.eu, the section Notice language describes a notice with several languages.
N.B
When preparing the minutes, the Publications Office took into account the responses of the participants to the survey at the start of the day. The Office noted that most participants mainly want to download the daily batch of all notices, in an automated fashion and then repurpose the data. They never use the TED website as such and they never make queries through the TED interface.
Comments from participants
@Everyone we're setting up a #slack workspace for anyone wanting to share ideas, code and examples on how to access and parse eforms. If you want to join, please just contact Alex at Spend Network to be added to the workspace.
For technical issues related to the SDK, participants are welcome to contribute and discuss on GitHub at https://github.com/OP-TED/eForms-SDK/discussions