3. Pipeline Data Exchange Format (PDEF) Specification¶
This specification is intended to gather the agreed requirements which shall be implemented in the design of PDEF.
3.1. Users¶
Users of PDEF shall be project engineers, subject matter experts, IT specialists and Data scientists working on subsea pipeline technical matters.
Users may come from any Company of the value chain : operators, contractors, manufacturers, service companies, etc.
3.2. Scope¶
The PDEF shall include:
data to be exchanged with others.
technical data only, and not include data related to individuals, organisations, administration, project control, contractual or commercial aspects of a project.
3.3. PDEF Dataset¶
The overall data required to be exchanged shall be called a PDEF Dataset.
3.4. JSON¶
The JSON language shall be used to write valid PDEF Dataset.
Note
JSON is a widely used text based data-interchanging format. It is defined as a small set of structuring rules for the portable representation of structured data. JSON specification is part of ECMA International, an industry association dedicated to the standardization of information and communication systems.
JSON is simply a text based data representation. It is not a programming language. Most programming languages can read, interpret and write JSON.
3.5. JSON Fragment¶
A series of JSON statement shall be called a JSON fragment.
3.6. JSON Document¶
A series of JSON Fragment assembled together shall form a JSON document.
Note
A JSON document can be read or written in a text file using any text editor.
3.7. PDEF JSON Document¶
A PDEF JSON Document shall be a JSON Document that follows the structure and convention agreed and gathered during the PDEF project and reported in the PDEF Documentation.
A PDEF Dataset shall be written in one or several PDEF JSON Document.
3.8. JSON-Schema¶
JSON-Schema shall be selected as the technology enabling the definition of a meta-model that describes explicitly :
the definition of the variables, both in terms of data type (e.g. “number”) as well as in terms of business vocabulary (“nominal outside diameter”),
the objects those variables shall refer to, defined in business terms (eg pipeline, anode), their structure and associated documentation,
the relations between the objects and how those relations shall be provided,
the overall structure of those objects, relations and variables, readable by humans as well as computer code.
Note
JSON Schema is a JSON media type for defining the structure of a JSON Document. JSON-Schema is intended to define validation, documentation, hyperlink navigation, and interaction control of JSON data.
At the time of writing, the JSON Schema specification is an Internet-Draft of the Internet Engineering Task Force (IETF). JSON-schema is already widely used over the internet to allow interoperability between systems and applications.
JSON-schema is widely supported by various code languages, software, applications, database management system and platforms.
3.9. PDEF JSON-Schema¶
the PDEF JSON-schema shall specify the structure, type and documentation of a valid PDEF JSON Document.
The PDEF JSON schema shall be published at a unique URL to allow Users to explicity refer to it.
To be valid, a PDEF JSON Document shall therefore validate against the PDEF JSON-Schema published at this unique URL.
3.10. Example PDEF Dataset¶
To allow for testing and better understanding, an example PDEF Dataset shall be produced.
The example PDEF dataset shall be a valid PDEF JSON Document.
The example PDEF dataset shall be published at a unique URL to allow Users to explicity refer to it.
3.11. Documentation¶
The PDEF JSON schema shall be documented thorougly.
All objects and variables shall be documented explicitly.
Examples shall be included in the documentation to facilitate understanding and usage, typically using snippets.
The present specification shall be published as part of the Documentation.
3.12. PDEF Data Model¶
The objective of PDEF shall not be to set a one-size-fit-all data model.
PDEF shall specify an interchange format so that the data can be exchanged between Users.
Because PDEF intends to guarantee machine and human interpretable exchange of data between Users, a certain degree of data structuring and modelling must be implemented in PDEF. This shall include a business data model (eg pipeline, project, linepipe, coating …), as well as a physical representation under the form of a PDEF JSON Document.
For this reason, the choices made for the implementation of PDEF shall be primarily judged on their ability to serve the purpose of data exchange.
Note
Most data models shall remain the property and responsability of the Users. They shall be tailored to the specific business objectives of those Users.
Strictly speaking, the PDEF data model can be used for business related applications other than data exchange. However, it is not the primary objective of PDEF. From a user perspective, developing a specific data model driven by the business case, capable of reading input and writing output in PDEF format might be a much better choice, depending on the context.
There is no doubt that other model design would serve those business cases much better. But such user specific objectives are not the scope of PDEF.
3.13. Database Technology¶
PDEF is not a database.
PDEF shall be limited to setting the standard format of exchange between information systems.
The database technology shall be selected by Users and will differ.
Various user cases shall be covered, for example:
a Users with no database management system shall be able to read and write data in valid PDEF JSON Document.
an organization with a series of database management system based on relational models (eg SQL) shall be able to extract the information from its database and produce valid PDEF JSON Document. These documents must be readable by another organization using another type of database management system, eg NoSQL, graph or document.
Users will implement proprietary data models dedicated to their specific needs and applications. The most appropriate database management system shall be selected accordingly.
Note
Strictly speaking, PDEF JSON Document could be used as a physical data base layer, although this is not the objective pursued in PDEF.
3.14. Scaling¶
The PDEF JSON Schema Design shall allow for any PDEF Dataset of any size, ranging from a few to many objects and variables, typically required to describe a complete set of pipeline facilities.
Considering that PDEF is not primarily intended as a Database Technology, performance shall not be the primary design criteria. However, transmitting large amount of data shall be made possible using the PDEF format.
3.15. PDEF JSON Schema Design¶
3.15.1. General¶
To allow evolution and maintenance, low-coupling and extensibility shall be implemented in the PDEF JSON Schema Design.
A flat and structured design shall be selected.
Nesting shall be avoided for performance, ease of use and maintainability.
Note
The PDEF JSON Schema Design is inspired from NoSQL document type database system.
3.15.2. Base JSON Fragments¶
Wherever repetitive patterns are to be used, they shall be normalized and a convention shall be set.
Existing conventions and current business practice shall form the basis for designing Base JSON Fragments.
Base JSON Fragments shall become the building blocks of the PDEF JSON Schema Design.
3.15.3. Root Object¶
A valid PDEF JSON Document shall be a JSON object, gathering standardized Collections of Objects, see below.
3.15.4. Collections of Objects¶
A valid PDEF JSON Document shall be organized in series of collections gathering objects.
Collections shall group object instances of the same class. For example, the bare_pipes collection shall be the series of all the instances of BarePipe (FV) objects necessary to be exchanged.
It shall be possible to use PDEF to exchange limited subset of data, as well as complete sets of inter-related data.
For example, one user case might be to exchange only some limited linepipe related data, while another use case might be to exchange the full technical data related to one project, or several projects.
PDEF shall include one or several distinct collections to be provided within the same PDEF JSON Document.
Only the Collections standardized within PDEF shall be accepted in a valid PDEF JSON Document.
3.15.5. Objects¶
Objects shall group variables defining a business entity.
For example, the bare_pipe object shall describe a specific & unique bare pipe physical object (ie more or less a 12m long steel tube used to make a section of pipeline).
An object is made of Key-value pairs, also known as attributes. Each key-value pair shall be called a member.
Object nesting shall be minimized.
Objects shall maximize the use of repetitive patterns normalized in Base JSON Fragments (such as eg physical measures, geo-reference…)
3.15.6. Object type¶
Objects are normalized in types, also known as class.
For example, the description of a bare pipe physical object shall be normalized in a BarePipe (FV) type.
Object type shall be specified within the object under a dedicated member, pdef_type for example.
3.15.7. Object ID¶
All Objects shall have an identification member, called Object ID.
This ID shall be unique, within the context of the PDEF Dataset transmitted.
To avoid naming conflicts with existing database management systems, pdef_id shall be used as the key of the Object ID.
PDEF shall validate any string of characters format for Object ID.
PDEF shall recommend to use version 4 UUID.
Note
A Version 4 UUID is a universally unique identifier that is generated using random numbers. The Universally Unique IDentifier (UUID) URN Namespace specification (RFC 4122) is maintained by the Internet Engineering Task Force (IETF).
3.15.8. Relations¶
Relations shall be used to refer other Objects instances contained in the same PDEF Dataset.
Relations shall be explicit and optional.
The schema design shall allow for embedding and referencing.
Embedding can be selected for one-to-few relations.
Referencing shall be using the unique Object ID key. Referencing can be used for one-to-many and many-to-many relations.
3.16. Conventions¶
3.16.1. Dataset Version Control¶
The purpose of PDEF is to set a standard for exchanging data between stakeholder.
As a result, setting a universal version control and management of the data is not within the perimeter of the PDEF project.
Version control of the data remains the responsability of the Users.
Note
The Meta Data feature can be leveraged to capture version stamping and other description to allow the emitting and receiving parties to explicitly specify the version or versions of the PDEF Dataset used.
The parties shall however agree on the schema of the metadata and how their version control management system interplay with such schema.
3.16.2. PDEF JSON Schema Version Control¶
For forward compatibility and avoid inter-operability issues, the PDEF JSON-Schema version shall be included as a member of the Root Object. The PDEF JSON Schema Version Control will be mapped using the pdef_version key.
PDEF JSON-Schema shall be versioned using an issued year-month stamping convention.
3.16.3. Required Members¶
JSON-Schema allows to specify that some members are required to be provided as part of the JSON Document. This is specified using the required key in JSON-Schema.
In PDEF, by default the members shall not be required for the following reasons:
PDEF intends to normalize the form of the data to ensure exchange, but not the scope which remains the responsability of the user.
the data is not necessarily available at the time of writing the PDEF JSON Document. It is considered to be a better practice not to have the member than to carry series of empty members.
if a context calls for some data to be required from other Users, such requirement shall be specified within this context, which is not the purpose of PDEF.
Exception to this rule applies to properties that are essential to understand the model, for example Object ID and Object type.
3.16.4. Additional Technical Data¶
For practicality reason, it shall be possible to add data not strictly referenced within PDEF. However, such Additional Technical Data shall be unambiguously set apart from the structured PDEF data, eg using a dedicated AdditionalData member.
The AdditionalData member shall be an object whose members can be any structure defined by the Users.
Additional Technical Data shall be accepted for all Objects.
However, it shall not be expected by the sender that the additional_data members will be parsed and interpreted by the receiver.
3.16.5. Additional Properties¶
Additional properties shall not be allowed in PDEF. Additional Technical Data shall be used for the purpose of conveying technical data not covered by the current version of PDEF.
Note
Additional properties are extra members that can be added to a JSON Fragment, in an unstructured way. The JSON-Schema specification offers the possibility to allow or restrict the use of additional properties. If such allowance would be granted, Users could add extra key-value pair into the PDEF JSON Document which would remain valid. In the opposite, the restriction does not allow the addition of extra key-value pair.
This restriction guarantees that the data passed to a partner is fully explicit for the receiver. If Users need to exchange data not implemented in the current PDEF JSON-Schema version, two options are possible:
a dedicated context based schema can be implemented between the users,
or it can be requested to update the PDEF JSON-Schema to include those missing properties, thus extending the model for all the other Users.
Note
Exception to this rule applies to Meta Data and Additional Technical Data which can be populated with any user defined member.
3.16.6. Measures and Unit of Measure System¶
Whenever a physical measure is to be provided, there shall be no ambiguity with regard to the unit of measure used to cast the value.
At the same time, it is recognized that various unit of measure may co-exist in the same PDEF Dataset, according to the business practice and context.
A valid PDEF JSON Document shall not require uniform or even consistent unit systems, but shall rather be based on explicit declaration of the unit used to report the measure.
3.16.7. Geo-Referenced Data¶
A valid PDEF JSON Document shall allow for geo-tagging objects.
The geo-referencing convention to be implemented shall capture positions in 2D and 3D using coordinates in a coordinate reference system.
Whenever a geo-reference is to be provided, there shall be no ambiguity with regard to the coordinate reference system used to cast the geo-reference.
At the same time, it is recognized that various coordinate reference systems may co-exist in the same PDEF Dataset, according to the business practice and context.
A valid PDEF JSON Document shall not require a uniform or even consistent coordinate reference system, but shall rather be based on explicit declaration of the coordinate reference system used to report the geo-reference.
3.16.8. Meta Data¶
Metadata are data about the data. It describes status, version, context, stakeholders etc associated with the PDEF Dataset.
It shall be possible to include meta data in a valid PDEF JSON Document.
The meta data shall be recorded, but clearly identified as such and not mixed with the technical data. A dedicated mechanism shall be implemented to capture meta data.
It shall be possible to include meta data at various locations into a valid PDEF JSON Document, although it shall not be required.
Metadata shall be accepted for all Objects.
The MetaData instance shall provide metadata related to the object it is embedded into.
The MetaData instance shall be populated with any user defined members, but it shall not be expected to be interpreted or maintained by the receiving user.