eForms Technical Workshop

 

The eForms Technical Workshop took place online over two half days from 11 to 12 May 2022. 

The minutes include replies to questions addressed live during the webinar. These have been reviewed and may have been regrouped or modified for this written version to allow for a more cohesive and complete response. 

The recordings of the live sessions are available in the Agenda section along with the PowerPoint slides of each presentation (where available). 

  

11 May 2022 – Day 1 

Welcome and introduction – Hilde Hardeman & Maria Manuela CRUZ - Publications Office of the European Union 

Ms HARDEMAN, the Director General of the Publications Office (OP), introduced the topic of the workshop stressing the importance of eForms for better eProcurement data. The eForms project is important, ambitious, complex and demanding for the Publications Office and eSenders alike. In this joint endeavour, OP shares its progress, resources and documentation and welcomes in return eSenders’ input, questions and concerns.     

 

Overview of eForms: timetable, applications and workflows – Karl Ferrand - Publications Office of the European Union 

Mr FERRAND gave an overview of eForms implementation, starting from 2019 – and the eForms implementing regulation – to the current status of OP developments. He went on to remind some points about the eForms schema: the UBL common components used (prefixed cac, cbc) and the eForms specific elements added (prefixed efac, efbc). He also discussed the scope (40 eForms plus 5 other forms), Business Terms and Fields (BT, OPP, OPT).  

Mr FERRAND outlined the eForms rules, i.e. the different types of rules and presented the Central Validation Service (CVS) as a “single point of truth” used across applications for validating eForms notices.  

He continued with the transition timeline for the various systems/applications and gave an overview of all future TED systems at the Publications Office:  

• eNotices2 (website and API and their respective functions),  
• CVS and TED Viewer (as API only),  
• TED API 2022,  
• TED Monitor 2022 and  

• TED and TED 2.0.  

He then presented the various notice statuses available within eNotices2 and through the API as well as the “Preview” environment that OP will progressively make available to help eSenders develop and test their own applications. 

Information about the Preview environment and the status of each application is available at https://docs.ted.europa.eu/home/eforms/preview/.

Mr FERRAND presented the main points of the 2022 amendment to the eForms regulation that is to apply from 14 November 2022. He informed that OP will start implementing the changes even before the amendment is published. He concluded with the next steps (including SDK version 1.0.0) and issues of eForms implementation.  

  

The following questions were addressed: 

• Could you elaborate on the transition period? Until when TED notices can be sent in the current formats. Are TED notices and new eForms notices displayed in parallel on the TED portal? 

Answer: The last day when OP will accept notices in the current TED schema format will be 24 October 2023.  

TED notices and eForms notices will be displayed in parallel. On 14 November 2022, TED will display the new eForms while it will continue showing all notices submitted for the past 10 years even after 2024. The current reception (eSentool and current eNotices) and production systems will be discontinued at the end of 2023.  

• Can you comment on the publication schedule? What is the foreseen transition time between statuses submitted / publishing / published? Will there be a more or less fixed publication scheme as there is today? 

Answer: The release calendar on TED includes the dates on which there will be an issue of the OJS. The OJS will continue to be published from Monday to Friday apart from certain public holidays.  

When you submit a notice, you can select a preferred publication date or you can ask for publication “as soon as possible”. During the initial stages, we will probably aim at publishing within five days but it may be technically possible in the future to publish sooner provided the notice has passed automatic validation and the manual checks for lawfulness (e.g. whether an entity is entitled or not to publish in TED, or whether certain procedures are to be announced in TED).  

• Could you explain CM and EM fields? Are they mandatory when visible under certain conditions? 

Answer: EM is mandatory if it exists, i.e. if the Contracting Authority has the information, they should fill it in. CM is Conditional Mandatory, i.e. mandatory if certain conditions are met.  References to CM and EM are not part of the annex to the Regulation; they are included in the so called “extended annex” Excel sheet that was provided for information and clarification purposes. 

• Will CM (Conditional Mandatory) also be taken into account in the adaptation of the regulation? 

Answer: The regulation does not include reference to CM. New fields that have been introduced will also be either Optional or Mandatory. Information on Conditional Mandatory will be specified in the updated extended annex.  

• Are the rules for CM documented in detail? If so, where can one read about these conditions? 

Answer: The conditions are visible in the Schematron rules as well as in the new eForms expression language, efx-grammar, in a more readable form.  

• Do you foresee to create custom profiles in CVS so that you can check if the submitted notice is valid according to the national tailoring rules? 

Answer: No, our developments do not take national tailoring into account. The implementation of the eForms regulation in national legislation will not be taken into account for the CVS.  

• Would it be possible for the Publications Office to at least act as a knowledge resource for national tailoring or will each software developer need to get in touch with national authorities for each country that our software will need to support? 

Answer: OP cannot assume this role. For commercial eSenders who are not working directly with the ministries and are operating in multiple countries, this has been raised with the EXEP committee of the Commission. The Commission would have to ask every Member State to make this public, but there is no central source for now where Member States publish this information. 

• How will the fields that are generally optional but that a given Member State decides are mandatory for that country be handled with regards to validation before publication? 

Answer: This relates to tailoring, which can only make rules tighter at a national level. If certain optional fields become mandatory at a national level only, OP will continue to consider them optional.  

• There are a lot more fields in the SDK than in the Implementing Act. Is there a need to address this added content in the national tailoring? 

Answer: The added fields are there for technical reasons, and should not concern you. You should specify instead which business terms should be mandatory. 

• What is the legal value of the “5 other forms”? 

Answer: The eForms regulation implements the Procurement Directives and has 40 eForms. The 5 other forms result from other EU legislation that imposes publication of specific information in the OJ S: 

• T01, T02: regulation 1370/2007 (public passenger transport by rail and by road) 
• X01, X02: business registration (European economic interest grouping and European company/cooperative society) 
• CEI: call for expression of interest (by EU institutions) 

The voluntary forms preliminary market consultation, contract completion and the three below threshold forms currently are not foreseen in the EU legislation and have no legal value, so we are not implementing them at this stage. These forms are shown as E1-E5 only in the extended annex Excel sheet. 

We do have forms E1-E5 in the schema (not in their final version), but they will only be fully implemented after eForms are mandatory in October 2023 and their use, which currently has no EU legal basis, will be optional. However, the Publications Office is aware of the importance of these forms and will try their best to have a stable schema implementation by the end of 2022.  

• It seems then that TED XML has to be kept alive for the T01 and T02 forms. Unless there is a 100 % voluntary migration to eForms for these forms? 

Answer: Our intention is that T01, T02 forms will also close in October 2023 with the current schema. If you are sending these forms, you will have to adapt to the eForms schema, as migration to the new schema is mandatory.   

• What is the meaning of “not published” status? For example, what is the difference between "Submitted" and "Not published"? Will there be reason codes for “Not Published” notices?  

Answer: A user working on the user interface of eNotices2 will be able to see the following notice status: 

• Draft: The notice is being drafted. 
• Locked: The notice is being edited. 
• Validated: The notice is validated and ready for submission. 
• Submitted: The notice is successfully received, validated and sent to OP (received by TED-Monitor-2022). 
• Published: The notice is published online on TED. 
• Stopped: Publication of the notice was stopped by the buyer/ eSender before publication and the request was accepted. 
• Not published: The notice was received but not published on TED.

If a notice is rejected due to manual lawfulness checks, the notice would have the status “not published”. A notice that is in status “submitted” will either be “published” or “not published” unless you stop it in between.  

From an API point of view, a notice that fails the CVS check will not get submitted. The CVS validation will give a validation report, which will detail all the reasons why a notice did not get through. Rejection due to lawfulness manual check will be communicated via email.  

• Will a lawfulness "fail" also be communicated via the API or only via email? This is quite important for eSenders. 

Answer: The API will tell you that there is a lawfulness warning when you submit or if you validate before submitting, so eSenders will be warned that there might be a rejection. If a notice is manually rejected due to lawfulness, it will go into status “not published” which can be queried through the API. We will not actively inform the eSender, but the contracting authority will receive an email.   

• Will there still be checks as to the language in which the notice is written -> NOT_PUBLISHED (Reason = Wrong language)? 

Answer: No, because we will no longer check the languages of each text field. “Not published” at this stage refers to not lawful.  

• Does the TED XML to eForms schema converter service generate completely valid eForms, or is it for input only? 

Answer: It is not possible to fill in all the mandatory fields from a current TED XML in an eForms XML, so by definition the converter will convert as many fields as it can, but it will not be a valid eForms notice. This "partial" eForms notice will still need to be completed and checked in the eSenders’ systems.  

• Where can I find the visualisations of all the eForms?  

Answer: The “eForms notice samples” in PDF were published on SIMAP on 23 July 2021 and the “updated samples of eForms notice visualisations” in HTML were made available on 11 May 2022. 

Some new view-templates are also available in the SDK as a definition of how an HTML/ PDF should appear.  

• Will there be XML samples that include all fields (according to the form and its conditions)? 

 Answer: We have some examples and we will try to build up samples as we progress, where we try to fill a maximum coverage of fields. Not all fields can be filled, as some fields are conditional.  

• Do you see any limit for XML size? Based on lot number format, there could be 9999 lots, which makes a file around 100 MB big. 

 Answer: Currently there are no limits, but we may have to put them in place if our systems cannot cope.  

• In your manual checks, will you continue to check if there is a mix of languages in the notices? 

Answer: OP will not be able to check anymore the text in multilingual fields against the language. The only manual check will be for lawfulness. 

 

Demo of front-end eNotices2 application – Enrico Campanella - Publications Office of the European Union 

Mr CAMPANELLA presented the front-office features of eNotices2, demonstrating the following improvements in “My forms settings”: 

• Forms settings, which include: 

Applicable legal basis 

Notice types 

Form language options 

Default currency 

Optional/ Mandatory fields 

Field Confidentiality  

• Field default values 

• Main buyer settings 

As there are many organisations participating in procurements, some being recurrent, the Address Book was enhanced with the options of other buyers, tenderers, natural persons and service providers. These addresses can be saved and imported in the notices in the form-filling tool.  

Mr CAMPANELLA continued to demonstrate the improvements in the form-filling tool, including the search function per Lot in the “notice sections” and the toggles for “warnings” and “errors” once the CVS is integrated with eNotices2. He continued with the improvements in the “Procedure” section, which remove all optional fields in the form-filling tool and the new feature of character limits displayed in certain fields. He demonstrated the display of codelists fields as well as the hierarchical codelists for CPV and NUTS codes. 

 Mr CAMPANELLA continued with a demonstration of the new actions available in “My notices” section for front-end users and the set of rules and profiles implemented for “My workgroups”. 

 

The following questions were addressed: 

• Is the user interface dynamically built based on the eForms SDK or are the rules (legal combinations) hardcoded? 

Answer: We have some possibilities to tweak the groups and the sections in the notice type definitions (which is another internal part of the application) but everything related to the fields in the form-filling tool comes from the SDK. 

• Will defining "My forms settings" in eNotices2 user interface also define rules for notices sent by API? 

Answer: Defining “My form settings” works for the user interface (UI), so it refers to the front-end only. The user settings can be used either by the Workgroup manager or by the administrator of a structured organisation when it will be available, but it is not related to the APIs. 

• Is 2000 Lots a hard limit or one that only applies in eNotices2 UI? 

Answer: 2000 Lots is the technical limit implemented right now, as a higher number would make a notice quite difficult to render.  We may also expect a Contract Award Notice with 2000 Lot results and as many tendering parties and tenderers as necessary, which would result in a huge amount of information to be transmitted. 

• Am I right in my understanding that one Lot (BT-13713 - Result Lot Identifier - not repeatable) could not be split between more than one contracts (BT-150 - also not repeatable)? 

 Answer: It will be possible to associate a Lot result to more than one contract.  

The Lot result is repeatable, whereas the result of the notice is not. The annex to the regulation refers to Business Groups, which, if they are repeatable, can therefore contain a repeatable set of Business Terms. 

• Is it possible to ask OP to turn off the possibility of buyers from a certain country to directly use eNotices? In doing so all notices would have to go through an eSender's interface, which would ensure conformity with national tailoring. 

Answer: This is not foreseen for the moment and would need to be discussed if the need arises, e.g. when there is an obligation at national level to use eSenders. 

• Could you elaborate on how "importing" data into a notice is coded? 

Answer: When a notice is imported in eNotices2, the XML is turned into a JSON by the application, and we show it in the form filling tool if the schema is valid. 

When an organisation is imported from the eNotices2 address book, each user can define a list of addresses and an address contains a piece of a JSON notice (a snippet). That snippet is created when the user creates an address based on the snippet type (equivalent to notice type) defined in the application.yml + information from the field repo. The snippet type also defines the sdk-version and the group id (GR-...) where this kind of address should be inserted in the actual notice.  

Each address also has a 'role' assigned to it defining the fields in the notice that need to point to this address when imported (ex OPT-300-Procedure-Buyer should get ORG-xxx corresponding to main buyer).  

There are two ways addresses are imported: at the creation of the notice, we import the main buyer (inherited from the form settings) and in the form filling tool when clicking on the “add address”. When we import an address, we locate the correct group in the notice, create a new instance and fill all the subfields recursively in the group instance with the value from the selected address. And we also set pointer to the new address in the fields defined in the role. 

• Are all text fields in eNotice2 single-line only? 

Answer: We can define in the notice type definitions if we want a text field of one line or a text area and we plan in future to have a feature that increases the size of the text field to a certain limit. 

• Can I send a notice to a specific Workgroup via Web Services? 

Answer: There is no service that currently allows that. You can send notices via the Web Services API to the application and to CVS and TED Monitor. The notice will be stored under the “My notices page” but it is not possible to send notices to a Workgroup via Web Services; only via the UI. 

• Do I understand correctly that the topic of Workgroups does not play a role if a user uses a procurement platform and therefore does not see the eNotices2 interface?  

Answer: If you or your users are not interested in using eNotices2, there is no obligation to do so. The only obligation will be for eSenders to create an eNotices2 account in order to pair it to their API key. 

• Will every form update end in an SDK update? 

Answer: In fact, every time eNotices2 receives SDK updates, as with any other application. SDK updates will have to be imported, and the new fields/ definitions/ conditions will be reflected into the form-filling tool. We will also need to tweak the notice type definitions.  

• Will we have to use the TED platform only in the future? Or can we still use the regional/national e-procurement platform (foreseeing a netmask that sends eForms to TED, as it happens now)? My question is related to publication obligations at national level. 

Answer: At OP, we have to make available a web platform to submit notices, which is what we have done with eNotices2. We are also sharing the platform with you, mainly for inspiration purposes. At national level, if there are eSenders and an obligation or the possibility to use eSenders, the contracting authorities can very well keep on using them and their platforms in order to prepare and send their notices. Like the current eNotices application, eNotices2 will be available for buyers who do not go through an eSender, by choice or by obligation. 

• Can you elaborate on how important data is coded in the notice? eNotices2 allows you, for example, to import Organisations from the address book when you are filling in a notice. How is this done and how can the code remain dynamic? 

Answer: What is saved in the address book is the actual address of an organisation. In the form-filling tool, you see this organisation as generic, and you have to pair it with a role in another part of the notice. The technical identifiers (like ORG-xxxx) do not have a lifetime outside the context of the notice and every time you import an organisation from the address book into a notice, it will have a new identifier.  

• Can you elaborate on how to pre-fill the organisation data and the fields in the eNotices2 web application? How will the Schematron validation errors be presented to end users via the user interface of eNotices2? 

Answer: eNotices2 will receive the validation report and the report will be embedded in the form-filling tool. The user will be able to see on the left of the screen some text boxes with errors and warnings and click on them to be directed to the errors to correct. The text boxes will contain the error text from the validation report and the user will have the possibility to sort through the notice to see only the warnings and/ or errors. 

The validation report gives you an XPath to the location in the XML where the error was detected. Then it is up to the application to use that to point the human user to the right place that needs to be corrected. We are working on adding supplementary information (as a relative XPath) to go down to the most granular level, for cases where the location does not already indicate this. In particular, if a field is missing, the location points to the container in which this field should be, so the additional information will point to the location where this field is supposed to be. 

• So, If I understand correctly, the exact location in the form (also considering a field in "lot X") will be documented later on in the SDK? 

Answer: The results of Schematron validation in CVS would give you this information in SVRL. 

• Does location equal XPath? 

Answer: Yes. 

 

Using eForms APIs – Enrico Campanella - Publications Office of the European Union 

Mr CAMPANELLA presented the steps to accessing the API and the Swagger documentation (https://enotices2.preview.ted.europa.eu/esenders/swagger-ui/#/Notice) which is still in progress; the URLs and the parameters will be reviewed. He then showed Curl command examples for GET and POST requests and examples of responses.  

 

The following questions were addressed: 

• Will the TED eNotices2 account be different from the EU Login account? 

Answer: You will need to have an EU Login account to be able to connect to eNotices2 and create an account. The same EU Login account will be used to log into TED API to manage your API key. 

• Is the API key the same for test and live environments? 

Answer: There will be different API keys for the Preview environment and for the live production environment. 

• How do we create an account for us as an eSender? It doesn't seem logical to use my personal EU Login to create an eNotices2 account for our national eSender platform. 

Answer: It is not required to use your personal EU Login. You can create an extra EU Login for the platform and the eNotices2 account with the credentials of your choice, for example, using the email address of a functional mailbox used by your eSender application.  

• Does an XML need to be uploaded prior to validation? The validation API parameter is notice UUID. 

Answer: You will be able to call the CVS to test and validate a notice without having to submit the notice. 

• How long would the validation report remain available?   

Answer: You can call the CVS at any time to have the SVRL validation report returned to you.   

• Can I send an incomplete notice via Web Service-API and continue via eNotices2 UI?  

Answer: No, the notices have to be complete before they are submitted via API.  

• Is API already available to use now? 

Answer: Once the Preview environment is available, you can start using the API.  See the latest status at: https://docs.ted.europa.eu/home/eforms/preview/    

• If a CN is sent in current eNotices and then is followed by a CAN in eNotices2, do you foresee any issues?  

Answer: We do not foresee any issues. We will implement a way to connect the two in a document family. The partial converter will also be available, which allows some of the information from the TED-XML CN to be carried over into the eForms CAN.  

• How will the procedure ID be added when the initial notice was not an eForms notice? 

Answer: We do not plan to add the Procedure ID to the TED XML notices. The eForms notice will have a Procedure ID. A contract notice in TED-XML schema will not have the Procedure ID but a contract award notice in eForms will have the procedure ID and they will be linked together in TED through another Business Term, which we haven’t finished defining yet.  

• Will there be quota on the usage of validation service?  

Answer: We will control the use of Web Services. This will be defined at a later stage and at a technical level. 

• For multi-line fields, will there be any validation about breaking lines, special characters (like umlauts, quotes) etc. or will it continue as is?  

Answer: It should continue as is.   

• I can't find anything in the SDK 0.6.0 (fields.json or notice-types) about whether a text field should be single or multiline. 

Answer: This is not fixed, and it is related to the notice definition. We plan to review and rationalise the notice type definitions as the new SDK versions come out. In the notice type definition, it is a choice by the person who makes the notice. If a field is a description, it could be a text area, if it’s a title, it could be a text field. We have a json file, which at a certain point will be shared in the SDK.  

• What can you put in a text field at XML level? 

Answer: The validation rules only check the maximum size of the field. As long as the XML text does not exceed the maximum size of the field, it is acceptable.  

• In our national platform, we have a very similar approach where we make a little workspace for the user. Our business user would like to include things like procedure type in the preliminary filters. We see no way to get this information from the SDK without inspecting the JSON files of all the notice types to see if they include or allow the BT-105. 

Answer: The information of whether a field is allowed is in the eForms annex and the extended annex but there is no information if the values of the fields are allowed and in which notice subtype they are allowed. You would have to go through the fields repository to find if there are any conditions related to this. 

 

Overview of the Software Development Kit and docs.ted.europa.eu – Karl Ferrand - Publications Office of the European Union 

Mr FERRAND presented the major changes in the latest eForms SDK release and gave an overview of the TED Developer Docs: 

• SDK 0.6.0 
• New eForms Expression Language (EFX), replaces SpEL, allows more rules 
• New view-templates 
• Updated fields, rules, notice types, codelists, examples 
• New code in https://github.com/OP-TED  
• efx-toolkit-java: Java EFX expression translator to Xpath, used for Schematron 
• eforms-notice-viewer: sample application, uses EFX and view-templates to generate HTML from notices in XML
• ted-xml-data-converter: XSLTs to convert TED-XML to eForms XML  
• docs.ted.europa.eu 
• FAQs 
• Versioned with SDK 
• Documentation for each SDK component: schema, field metadata, codelists, etc 
• SDK roadmap: docs.ted.europa.eu/home/eforms/roadmap/ 
• Building metadata-driven applications 
• Documents ESPD and eProcurement ontology 
• eForms Preview environment 

 

The following questions were addressed:  

• Will the additional repositories be maintained after October 23?  

 Answer: The Git repositories will live for as long as we have eForms.  

• Schematron validation returns some XPaths; may I expect to find exactly same XPath from fields.json in order to identify my BT in the UI? 

Answer: The Schematron validation indicates the location of the failure and it is quite precise. For example, if the second lot in the notice is missing a title, the XPath will point you to the second lot of the notice. It will not be exactly the same XPath from the fields.json as the field is more generic, whereas the XPath you get from the validation corresponds to a specific instance or notice. However, you will be able to figure out what is the actual field that has caused the issue. In the future, we will also add a bit more information in that sense, as eNotices2 needs this to point to the exact element in the XML that has the problem. 

• If a BT is made mandatory, is it mandatory across all notices for which that BT is relevant? Or can BTs be made mandatory for some notices, for example under the classical directive, and not others, for example, Utilities or Concessions? 

Answer:  The latter. You can choose for which notice types the BT is mandatory or not and if you look at the regulation, you can see that certain BTs are mandatory in one directive but not in the other. At a national level, you can also tailor at that level of granularity. 

 

 

12 May 2022 – Day 2 

Building metadata-driven applications using the SDK – Ioannis Rousochatzakis - Publications Office of the European Union 

Mr ROUSOCHATZAKIS presented what is new since November 2021 as well as version SDK 0.6.0 of May 2022 and its components:  

• eForms Expression Language (EFX) 
• Notice View Templates for notice visualisation 
• Latest metadata (rules, translations etc.) 
• EFX Toolkit for Java 
• Notice Viewer sample application 

He went on to discuss the next steps, including SDK 0.7.0 – which will review the Notice type definitions and complete the EFX features – the Metadata Reference Documentation and what comes after the final version of SDK 1.0.0. 

Mr ROUSOCHATZAKIS discussed the takeaways from Day 1 of the workshop and went through the main challenges eSenders are facing or expect to face in eForms implementation. In particular, he discussed: 

• Metadata driven architecture (https://docs.ted.europa.eu/eforms/latest/metadata-driven-applications.html) 
• SDK versioning, major/ minor changes and revisions (https://docs.ted.europa.eu/eforms/latest/versioning.html) 
• Target date for SDK 1.0.0 by end of July 2022 (update: now planned for early August) 
• Integration/ Tailoring and examples of challenges that could be addressed through the use of the SDK 
• Complexity and deadlines 

He discussed the challenges OP is facing and how these have been addressed by what is bundled through the SDK. He continued with the challenges and considerations for creating metadata-driven applications and how to understand parallel versions of the SDK.  

Mr ROUSOCHATZAKIS stressed that the Publications Office is focused on creating the conditions for OP and eSenders to be able to work together in a sustainable and efficient manner, but that the responsibility of building eSenders’ applications and business processes and managing individual systems lies with the eSenders. OP is interested in understanding eSenders’ concerns and difficulties in order to be able to consider them and solve any underlying issues that can be addressed through SDK updates. He concluded with discussing the possibility of opening a GitHub discussion for purely technical matters.   

 

The following questions were addressed: 

• We 'want' to use eForms SDK but are not sure yet how. Where could we find learning resources on how to deal with the different kinds of dependencies when dealing with metadata-driven applications? 

Answer: We will provide information in the documentation as soon as possible. 

• Is it possible to provide a guide with abstract steps on how to use SDK and create our eForms?  

Answer: A starting point would be the challenges and our experience we have shared during the workshop as well as the information and examples of metadata driven applications in the documentation: 

https://docs.ted.europa.eu/eforms/latest/metadata-driven-applications.html 

• When will SDK 0.7.0 and SDK 1.0.0 be published? 

Answer: SDK 0.7.0 was released on 11 July 2022. The target date for SDK 1.0.0 is August 2022 and we don’t expect it to contain major changes compared to SDK 0.7.0.  

• The updated Schematrons are creating connections to localhost:8080, so there seem to be some missing architecture we need to cover today. 

Answer: Indeed, in the latest version of the Schematron we have added some new kinds of rules for Change notices that not only look at the notice that is being validated but also need to fetch via a Web Service the original notice that is being corrected/ changed and compare some values.

The variable for the Web Service URL is currently set to localhost. If you want to reuse these Schematron out of the box, you can ignore the file that contains them and do a kind of a standalone annotation. If you edit the entry.sch file and remove the reference to define and change notices, this will remove these rules and you will not have the connections to localhost. We will have to see if we can address this better in future SDK releases.  

• Will it be possible to self-host the Schematron validator or is JavaScript the proper way to validate forms before sending? 

Answer: You will have TED CVS available as a service you can call before sending the notice. ENotices2 (web an API) will also call TED CVS anyway and send you back a validation report when the notice is submitted. If you want to validate on your side, you can use the Schematron files to create your own validator. JavaScript would be for client-side, live validation to make the user interface friendlier to the user, so that the user can see validation messages as they type.  

• How can you (in the metadata) tailor codelist? We want to reduce the options in some of the codelists. 

Answer: SDK codelists contains the tailored versions of codelists marked by an underscore. How these lists are to be used by a field is indicated inside fields.json. When the field is associated with a codelist, the identifier of the codelist is the identifier of the tailored codelist. The idea is that you can load the SDK and override the SDK definitions with your own compatible definitions which you will store locally in a second file like the fields.json file and which will only contain your tailoring. You could use this either for properties like mandatory or for properties such as the codelist ID that a field uses. 

• Would it also be possible to conditionally switch between codelists using EFX-Syntax - since by now each field that uses codelists is connected to only one codelist? With reference to national tailoring it might be necessary to exchange codelists based on certain conditions. 

Answer: We will have some rules that switch between codelists and we will partly use EFX to add them for this before version 1.0.0. Different rules are expressed in different ways and we are not going to unify the way we express the rules by using EFX everywhere. 

• Is it possible to split fields.json by notice types? It is hard to find which field is connected to which notice type. Can we get some list where to split it? 

Answer: Currently the fields are only connected to notices through business rules. We understand the issue but we are not going to remove the file nor do we have the capacity to provide such a list, as a field is connected to a notice type in several different ways. We may think about introducing some alternative formats after SDK 1.0.0. 

• Is there an SDK for other languages, e.g. .NET, PHP, Java, JavaScript? 

Answer: Currently we have an implementation of the translator in Java. The parser generator that we use – ANTLR - supports generating parsers in Java, Go, Swift, C#, Python. We have used Java for our needs and we put the code out there on GitHub. However, it is practically impossible for OP to engage in solving individual requests unless it is a community effort, i.e. if some eSenders are willing to share their developments with others, but which OP will not be helping to create.  

• Would it make sense to provide ANTLR like an API, so we can use multiple languages? 

Answer: Not really. There would be no need to translate at runtime if all the translations are ready for use at the time of importing the SDK to your system. There is a folder in the toolkit called interfaces; there are three interfaces that you can implement and target the different languages with the same translator without having to rewrite it. You can also refer to https://docs.ted.europa.eu/ and the Javadoc inside the application to understand how it works. 

• A kind of integration cookbook for PHP would be great, and maybe building up a 'PHP Usergroup' to find the easiest and best solutions for the integration. 

Answer: If we go into some sort of community hosting, maybe those self-help communities could group by technology so that people using a certain technology could exchange information with each other. However, OP would not be involved. 

• I'm missing the option in GitHub to ask questions so that everybody is informed (instead of using email). This should be centralised. 

Answer: We do understand that there is a need for a forum, but OP does not have currently the capacity to follow up. If we decide to open discussions on GitHub for technical matters and for developers, OP will not be able to engage in every conversation or provide any information, especially in the first few months, but we will try to take into consideration the concerns and issues you raise.   

• Is there versioning in the eForms in addition to the SDK and will the API only accept specific eForms versions? In the past, you accepted R2.0.9.S04 until date x und after that date only R2.0.9S05. 

Answer: It will not be like in the past. Multiple active versions may exist in parallel so that every eSender has the time to switch to newer versions. There is only one version number and the API will accept all active SDK versions. We will only deprecate a version as a business/ legal decision or for important data quality rules we need to impose. The CustomizationID element within a notice only indicates the major and minor version which relates to the validation; only the major version dictates what application logic is needed to read the SDK. 

• On 14 November 2022, will you accept only notices in SDK v1.0.0? 

Answer: The new regulation has schema changes that probably won't be in version 1.0.0 of the SDK. It could be that by September we have a version 1.x, which has the new schema and the new eight fields as part of the changes that will come with this year’s amendment to the eForms regulation. 

• eForms-notice-viewer fails with a NullPointer exception for me. I followed the instructions and have the eforms-notice-viewer, eforms-sdk and efx-toolkit-java in my local Maven repository. I tried both the maven command documented on GitHub and running it from my IDE. Neither works. 

Answer: We only released the SDK and not the eForms-notice-viewer or the toolkit because we are still looking at it. The eForms-notice-viewer in the repository is not affected by the version of the SDK. The EFX toolkit will be updated every time there is a major version of the SDK or an improvement it its code, but has to be able to work with any version of the SDK.   

• What about performance of Schematron validation? It seems to be a heavy process and needs much time for initialisation every time you do validation. 

Answer: It depends on the actual Schematron engine you use, but they all have the initialisation phase where they load the Schematron and pre-process it so it can be transformed. As our Schematron is large, this can take some time.

However, the outcome of that is in memory and you can set up your applications so that you reuse this for the validation of every notice without having to go through the initialisation process for every notice each time. You have to be careful with how you use your Schematron engine keeping the outcome in memory. The number of rules and the size of the notice and number of Lots will also affect performance.  

• We are working with SDK 0.5.0; will that mean that we will need to use the stable version for the production operation? 

Answer: You will need to update to a newer version before you go into production. 

• Would a change of the eForms regulation like the one advertised to happen in November this year lead you to update all live-SDK versions or start a new slate, with only one live SDK? 

Answer: Technically, there is no reason to deprecate. Depending on the changes in the regulation, we may have minor or major (breaking) changes in the SDK. Significant business changes, such as making mandatory some fields that were previously optional, would force us to deprecate an active version after a transition period.  

• If business rules are currently tightened with TED XML, then the first step is set the rule to ERROR on the qualification endpoint and then at a later point in time on the production endpoint. What will that look like in the future or will something change here? 

Answer: Things will change because we will no longer have the qualification process and Qualification/ Simulation endpoints will not exist. There will be two environments, Production and Preview.

The Preview environment that we are currently setting up will be available to you indefinitely. Fields will not be set to Error/ Warning, as the version of the SDK dictates everything. When there is a new version of the SDK, it will first become available only in Preview environment as a transition period for testing before we publish it in Production and the transition dates will also be announced.

We are currently developing an SDK Management Service. You will be able to call an API service to see which are the active versions of the SDK (the oldest and newest version you can submit) at any given time.  

• Will there be another workshop or Q&A after 1.0.0 is released? 

Answer: The next eForms workshop will be after the summer and it is planned for 29-30 September 2022. 

 

The following questions were addressed to the participants: 

• If there are any eSenders in the audience that evaluated the option of using the SDK but rejected it, we would like to hear from you what was the deciding factor. 

Comment: The concept of the SDK and data-driven development is to my mind nearly the only way and I've been developing data-driven applications for the last 30 years.   

• How do other Member States plan to integrate eForms in their existing platforms?  Will they sync data between the forms and the platform and if so,  how? Do they plan to maintain the agnostic / dynamic approach of the SDK or will they rather hardcode the eForms in their platform?  

There were no comments from the participants. 

 

Ms CRUZ concluded the workshop by thanking all participants for sharing their questions, comments and concerns. She stressed that there is still a lot to be done and information needed and the team is doing their best to address these. She reminded that the Publications Office remains supportive to eSenders and that the communication channel established for the best support possible is through the TED Helpdesk. 

 

 

During the workshop participants answered some slido questions. 

Slido poll results