Mike Lamoureux's Blog

Linked Data Whitepaper published!

Wed, 05 Jun 2013 01:04:00 -0400

The following white paper was written on the topic of Linked Data in the enterprise, and includes the value proposition for to build internal and external Linked Data infrastructures using Semantic Web Technologies. The white paper is authored by myself and Dr. Charlie Mead, my colleague at Octo Consulting Group. A downloadable version can be found here.

——

Introduction

Organizations of all sizes struggle with the problem of data sharing and interoperability across system and organizational boundaries. A unified view and ability to analyze distributed, yet inter-dependent data, is critical to both effective decision-making and increased situation awareness of any organization. Most organizations employ one-off integration strategies that not only increase the complexity of the organization’s integration fabric, but also do not effectively address the problem of data sharing among systems. Furthermore, these integration strategies typically tend to share only raw data rather than data plus context, and context is critically important in understanding meaning. In other words, lack of is a context / metadata is a significant barrier to interoperability.

One solution to these interoperability problems is to use semantic web technologies. Once these technologies have been applied, it opens a broader level of interoperability through the application of “Linked Data” principles. Linked Data is about using web technologies to expose, share, and connect pieces of data or information that have been “tagged” with semantic markup. A linked web of data need not be a goal of just free Internet evangelists. In the “anyone can say anything about any topic” world of the Internet, with its noted absence of central decision makers or defined organizational policies, semantic web technologies are being successfully applied to integrate data. Within an organization that has both policies and decision makers, the adoption of semantic web technologies can return even greater data integration value. Organizations of all sizes are collaborating more with their partners, suppliers, and customers with data. But even within the “walls” of an organization’s own infrastructure, there are benefits to sharing information using the same technologies that are designed for the linked web. Doing so allows for staying current with the technology industry, using tools (many of which open-source) to create an enterprise of linked data to arm decision makers with better information.

This paper will argue on the need for computable semantic interoperability, the dream of a data scientist or executive that desires to dig into metrics and data with ease. This paper will also discuss how the tools provided in the Semantic Web can promote a linked data “web of information” within a private enterprise, or optionally exposed on the web in an open or semi-open manner.

Defining the Data Interoperability Problem

A lot of work is being done to address interoperability challenges via – SOA, Data Integration, Data Standards Creation, and more – but there is a fundamental problem with all of those approaches. IT System Interoperability can be defined as the ability of systems to exchange information, understand the information exchanged and subsequently use the information that has been exchanged across system boundaries. However, this definition leaves several embedded important details remaining:

· Exchange information – In order to exchange information, systems must exchange both the structure (syntax) and the meaning (semantics)

· Use the information – The use of the information may be different in each situation of interoperability. For example, the information could be used by a human, or another system, for the purpose of displaying, or calculating.

The most common approach to solving interoperability challenges is to apply “one-off” integration strategies that, by definition, have minimal reuse or scalability potential since they are context-specific. In addition, one-off projects often realize narrowly scoped, short-term successes for specific vertical slices of an enterprise, the enterprise as a whole usually realizes relatively little benefit. In fact, over time, multiple one-off integration “successes” often become enterprise liabilities rather than assets as the enterprise itself gradually morphs into a collection of vertical, brittle, siloed integration solutions built on incompatible technologies with specialized code that makes the ongoing cost of just maintaining – not to mention expanding – enterprise interoperability increasingly expensive and burdensome.

The need for Computable Semantic Interoperability of Information

Computable Semantic Interoperability (CSI) is the sharing of information semantics for the purposes of subsequent processing of the shared information by the receiving system(s). The need for CSI is a natural evolution as we move into beyond a digital age, to one that requires interoperability across structured data from different “linked” systems. Organizations that seek to accomplish this while avoiding the problems mentioned previously often tend to look to the development of standards that can be applied proactively to generate more reusable, scalable CSI.

A Standards-Driven approach is not working

With respect to the development of data standards, both top-down and bottom-up strategies have attempted to meet the requirements of the diversity of CSI challenges. Bottom-up strategies naturally create data silos that contribute to rather than solve the problems of interoperability. In a number of domains, top-down agreement of metadata elements, data vocabulary and data serialization formats have been exhaustively negotiated between stakeholder groups to cover multiple use cases. However, in practice, the top-down approach suffers from a number of problems including excessive time involved in producing a given version of a specific standard, and a focus on “edge cases” that often makes the standard cumbersome and difficult for all but a few to understand. In addition, because a given standard is, in fact, a design- rather than implementation-level artifact, actual implementations often prove to be overly difficult (and sometimes even impossible) or inconsistent across implementation instances. In turn, implementation barriers give rise to niche vendors who emerge based on their understanding of the complex details of the standard, and, in response, build proprietary tools to manage overly complex implementation details. In the end, the road to achieving value from the use of a given top-down standard is often long and costly, and never truly supports the initial vision of CSI that motivated its original development.

There can never be a single standard that is driven top-down that covers all CSI instances. The development of multiple standards generates another level of CSI challenges – interoperability betweenstandards.

“Cooperation without Coordination” - Expressing shared Semantics in a standardized way through the Tools and Technologies of the Semantic Web

Understanding the meaning of data through semantics is a constant barrier to interoperability. To overcome this, humans must read data dictionaries and make manual connections in order to get to a point where queries can be performed across data sets. Computable representation of semantics (meaning) is missing from a majority of I.T. implementations and data sets today. Furthermore, most data is not referenceable over the Internet or intranet, and is instead held hostage by proprietary methods of storing information within existing I.T. systems that discourages or obfuscates people from making linkages outside of their realm of view.

The need for expressing semantics through a computable mechanism is not a new realization. However, the drivers to encourage the use of the technologies supported by the World Wide Web Consortium (W3C) called the Semantic Web have finally materialized. By following the open recommendations of the Semantic Web, cooperation without coordination is possible for those participating in the Semantic Web, meaning that developers are allowed to build-bottom up without creating non-interoperable silos. Over the past fifteen years, the World Wide Web Consortium (W3C) has defined and refined a number of standards that satisfy each of the following components:

· Resource Description Framework (RDF): A consistent (and extensible) data model for definition of the semantics of individual information components and the relationships between them;

· RDF Schema (RDFS) and Web Ontology Language (OWL): A companion model for defining schema which is itself expressed in RDF, thereby combining the semantics of both data and meta-data in a single representational artifact (a “graph”);

· SPARQL Protocol and RDF Query Language (SPARQL): A query language that supports all the of the features of SQL (including the ability to query existing Relational Database Management Systems (RDBMSs) stores equipped with a standard “SPARQL endpoint”), and in addition, provides other features such as graph transformation and distributed queries, all based on ReSTful technologies.

It is important to emphasize the point that the Semantic Web tools and technologies do not solve the “purely semantic” issues that must be resolved by the stakeholders who have the shared purpose to interoperate. Rather, the Semantic Web allows these interoperability issues to be surfaced and resolved by people on an “as needed” basis. Bottom-up resolution of semantic interoperability challenges is possible because of the decentralized extensibility of the semantic web.

The Era of Big Data, Linked Data, and Open Linked Data

The same interoperability challenges discussed before also affects a collection of largely inaccessible data, either because the data were buried in free-text web documents (renderable in browsers and therefore readable by humans, but not suitable for automated processing of the meaning of the data), or because the data was published in disparate, incompatible formats. This reality has given way to frameworks of technologies for “big data” which allow for developers to more easily work with large, distributed sets of data and perform algorithms on data sets.

With the maturation of Semantic web, better systems can be built to capture semantic context. Additionally, the existing mass of data and meta-data often can be utilized using Big Data technologies such as Hadoop along with the Semantic Web. In fact, the Semantic Web was defined, designed, and developed to support the evolution of the current “web of documents” to a more powerful and useful “web of data.” Just as in the web of documents but now focused on data, data in disparate locations (i.e. residing on multiple servers and/or application files) are connected via computationally traversable links, the actual traversal occurring through the use of the established internet technologies of URLs and HTTP.

To this end, the W3C has defined a set four star rating system (see call-out box) for publishing Linked Data that have now been validated and implemented by a number of organizations involving multiple disparate data sets. One of the most notable instances of the application of these principles is the Linked Open Data Cloud (LODC)[1]. The LODC contains data from geographic, chemical, life sciences, and a number of other domains. As of September 2011, the LODC contained more than 31,000,000,000 data elements interlinked by 504,000,000 relationships. In addition, a number of large providers including Google, Facebook and many others have started embedding RDFa – the W3C’s data semantic markup standard – within their web pages, thereby enabling data within those pages to be automatically processed and linked to other data of interest.

What to Expect: Intranets of Linked Data supporting Business Analytics and other Semantic Web COTS solutions

While liked data was developed to transition the public web of documents to a public web of data, it also has much to offer in the creation of structured intranet of data for an organization. Organizations can (and should) create their own intranet of Linked Data using Semantic Web technologies, taking internally the four-star system for best practices of a linked web. It is important to note that although the W3C refers to the notion of Linked Open Data in which the linked data is contextualized within the entire Internet, organizations can adopt the practices and procedures involved in Linked Open Data to intranet/organization-specific contexts with equally powerful results. More than likely, an organization would have permutations of linked data, semi-open linked data (based on attribute or permission level), and fully open linked data, an outcome that is also supported by the Semantic Web.

The Linked Data approach builds on existing Internet technologies, and is compatible with legacy technologies (e.g. RDBMS data stores). The Semantic Web is not a rip-and-replace approach. Many analytics platforms can already use Semantic Web technologies in an enterprise. Information required for CSI use cases can be serialized between systems using technologies that enterprise and COTS are familiar with: XML, JSON, etc. It can be persisted in existing RDBMSs or be moved to optimized graph stores and queried in these same systems via the addition of a standards-based SPARQL endpoint (based on ReST). For new persistence stores, organizations may choose from the emerging product line of “NoSQL” databases that store native RDF graphs.

Custom Off the Shelf (COTS) vendors have begun to recognize the power of open source and open standards. COTS vendors are commonly adopting “Open Core” business models – that is the building of software on an open source stack that is easily portable to any commoditized infrastructure and software environment. Taking this trend further, COTS vendors that recognize the importance of giving full transparency to an organization’s data will begin to build products on top of Semantic Web technologies, and provide access to the data and metadata (in RDF and RDFS) through an exposed SPARQL endpoint to encourage inclusion in a linked data infrastructure.

The Semantic Web tools are no longer bleeding edge. Investing in applications of the Semantic Web to particular intra- and/or inter-enterprise CSI challenges provides a realistic, reusable, and scalable solution to the historically vexing problems posed by the requirements of CSI.

Conclusion

A Linked Data Strategy increases the value of data that has traditionally been confined to isolated silos. The Semantic Web’s tools and technologies support the development of Linked Data webs whose boundaries can be customized by each organization and its stakeholders. The combination of drawing on the lessons learned in past attempts to achieve Computable Semantic Interoperability and adoption of the Semantic Web-based Linked Data Best Practices provides a path to increasing the value of organizational data.

[1] More details on the Open Linked Data project can be found at http://linkeddata.org, a graphic can be seen at http://lod-cloud.net.

Not all cloud providers are the same

Sat, 01 Jun 2013 09:36:00 -0400

I wrote a blog post on this topic a long time ago, when I started my own company, and two years later the confusion still exists today. Cloud providers are very different from one another. Other than the bland guidance that NIST provides for the definition of the cloud, everything else just goes. Your organization might be large enough to have its own private cloud, but can it beat the competition from a fully featured public cloud?

Companies should not base their analysis of a cloud provider solely on the unit cost of their services. Benefits, and even cost savings can also come from features that are not supported by a particular cloud provider. For example, consider hosting your development servers in the cloud, but not having them run while not in use. This strategy will save you in the cost per unit for server uptime. While the development server is not in use, your company only pays for the configuration and storage of that server to stay in the cloud. Many cloud providers do not support this feature, and the few that do, can be more expensive for their core cloud services: bandwidth, server usage cost.

Control Indigo from Alfred via Workflow

Tue, 07 May 2013 23:07:00 -0400

I’ve completed the first beta of my Indigo Alfred Workflow, and posted the source on github. The workflow allows you to set devices, change variables (lights only for the moment), or execute action groups for a Indigo Home Automation Server. using Alfred The workflow dynamically loads your devices from Indigo, so it should work on anyone’s setup.

Looking for feedback and any bug discoveries.

Special thanks to David Ferguson’s example workflows, and Mark’s bash workflow helper function.

Sandy Point State Park

Sat, 04 May 2013 15:26:26 -0400

Sandy Point State Park

Alfred Workflow for Indigo

Sat, 27 Apr 2013 18:02:58 -0400

I wrote a simple Alfred workflow today for Indigo to turn on some key lights in my house. I plan on making it more generic and to load devices based on my input, for now it is static and done through a bash script. Here is the code:

#!/bin/bash

read a1 a2 «< “{query}”

server=”http://remote.home.mikelamoureux.net:8176”

lrl=”devices/Living%20Room%20Light”

fan1=”devices/Fan%20Light%201”

fan2=”devices/Fan%20Light%202”

fan3=”devices/Fan%20Light%203”

fan4=”devices/Fan%20Light%204”

if [ $a1 == “lrl” ]

then

if [ $a2 == “off” ]

then

curl -X PUT -d isOn=0 $server/$lrl

elif [ $a2 == “on” ]

then

curl -X PUT -d isOn=1 $server/$lrl

else

curl -X PUT -d brightness=$a2 $server/$lrl

fi

elif [ $a1 == “fan” ]

then

if [ $a2 == “off” ]

then

curl -X PUT -d isOn=0 $server/$fan1

curl -X PUT -d isOn=0 $server/$fan2

curl -X PUT -d isOn=0 $server/$fan3

curl -X PUT -d isOn=0 $server/$fan4

elif [ $a2 == “on” ]

then

curl -X PUT -d isOn=1 $server/$fan1

curl -X PUT -d isOn=1 $server/$fan2

curl -X PUT -d isOn=1 $server/$fan3

curl -X PUT -d isOn=1 $server/$fan4

else

curl -X PUT -d brightness=$a2 $server/$fan1

curl -X PUT -d brightness=$a2 $server/$fan2

curl -X PUT -d brightness=$a2 $server/$fan3

curl -X PUT -d brightness=$a2 $server/$fan4

fi

fi

Children’s inn gala

Sat, 23 Mar 2013 20:33:53 -0400

Children’s inn gala

Happy St. Patrick's Day!

Sun, 17 Mar 2013 15:02:24 -0400

Nice relaxing St. Patrick’s Day after a week of travel. It caps off three successive weeks of travel to Boston, New Orleans, and Bermuda. JoEllen bought be Murphy’s Stout and Chili for lunch.

Birthday girl (at The Fairmont Southampton)

Fri, 15 Mar 2013 18:33:08 -0400

Birthday girl (at The Fairmont Southampton)

Defining the Problem - Octo Consulting's entree into Semantic Web

Sun, 10 Mar 2013 16:53:51 -0400

Note: This blog post was co-posted at Octoconsulting.com, with the help of our Director of Health IT, Dr. Charlie Mead.

Octo set out a few weeks ago to figure out a path to solve a problem that is simple to explain but difficult to achieve. We call it a “End-to-End Use Case.” The challenge is to find a way to discover “semantically equivalent” data that has been collected in multiple studies. Traditionally, this is a relatively straight-forward task if done pre-study (although it requires considerable “top-down” governance), but very difficult once a study has been “designed” and executed. The inherent semantic difficulty of the task is made even more difficult by ‘non-semantic’ barriers including different physical data persistence and access models, and the wire-format exchange” serialization brittleness” of XML, the lingua franca for much clinical trial data exchange. Our work is based on the overarching thesis that “end-to-end Semantic Web-based representation of study meta-data and data plus data transport formats largely circumvents the non-semantic barriers, thereby allowing study stakeholders to focus on the core problem: interoperable semantics.

Octo Consulting and members of the W3C’s Healthcare Life Sciences Working Group have developed a concrete instance of our hypothesis in which study meta-data and data are represented using based on RDFS and OWL ontologies of the HL7 Model Interchange Format (MIF) — the MIF includes the HL7 Reference Information Model (RIM), data types and vocabulary bindings — and SNOMED-CT. Data transport will use an Semantic Web representation of the CDISC ODM (Operational Data Model) specification. SPARQL end points are used for data discover and analysis.

Nice hat (at Nick’s Riverside Grille)

Sun, 10 Mar 2013 13:08:47 -0400

Nice hat (at Nick’s Riverside Grille)

Frenchman Street, New Orleans

Wed, 06 Mar 2013 16:49:44 -0500

Frenchman Street, New Orleans

Photo

Wed, 06 Mar 2013 14:47:20 -0500

Waiting for Bill Clinton to show up

Wed, 06 Mar 2013 14:45:03 -0500

Waiting for Bill Clinton to show up

Bourbon street

Mon, 04 Mar 2013 22:12:42 -0500

Bourbon street

The huge and always impressive @HIMSS #HIMSS13 stage with...

Mon, 04 Mar 2013 08:46:15 -0500

The huge and always impressive @HIMSS #HIMSS13 stage with @Octo_JayS @octoconsulting

The very loud train that rolls through New Orleans at 5am reminds me of My Cousin Vinny #HIMSS13

Mon, 04 Mar 2013 06:39:58 -0500

The very loud train that rolls through New Orleans at 5am reminds me of My Cousin Vinny #HIMSS13

Photo

Sat, 02 Mar 2013 14:10:43 -0500

Back in a DC state of mind

Sat, 02 Mar 2013 12:31:19 -0500

Back from Boston. Good times with @alexchiulli and my colleagues from @octoconsulting and @w3c

Next is #HIMSS13 so heading to New Orleans tomorrow.

Octo Semantic Web Hackathon in partnership with the W3C Life Sciences

Mon, 18 Feb 2013 22:44:00 -0500

Note: This blog article was co-posted at Octoconsulting.com

Octo is thrilled to announce a Semantic Web three day “hackathon”, (February 19th - 22nd), featuring guest Eric Prud’hommeaux from the World Wide Web Consortium (W3C) . Eric’s role in the Semantic Web Health Care and Life Sciences Interest Group (HCLS IG) is why we called upon him. While Semantic Web is a non-domain specific technology, we are deeply interested in how Semantic Web technologies can assist in the areas of healthcare and health research. We are striving to accomplish a number of objectives this week, some of which include:

Familiarize our consultants and technical staff on our use case - Developing an “End to End” solution for health information trails (See Dr. Mead’s upcoming blog post on “Defining the Problem”)
Develop a robust talent in Semantic Web for Octo’s brightest solutions team of Architects, Developers, and Management Consultants
Familiarize Octo with many of the open source tools in the market and build our own environment for Semantic Web tools Research & Development in our “Octo Labs” cloud environment
Develop additional materials for technical training sessions on Semantic Web technologies

Look for blog posts this week from our Octo team on the ground, featuring our perspective on interesting topics such as, how Semantic Web and Legacy IT are integrated,an update of the current progress of healthcare interoperability standards in Semantic Web, a recap on our daily progress at the “hackathon’, business intelligence and visualization, and much more. To get in touch, tweet us at @mlamoure or @octoconsulting with questions, comments, and thoughts. Wish us luck!

Objective-C

Sun, 10 Feb 2013 15:09:15 -0500

I now have much greater respect for Apple and iOS developers… Objective-C is some crazy stuff. Finally have a functional iOS app though, thanks to Lynda.com.

Note about my project: I’m creating a new remote control interface for my indigo server. Coming along well so far. I’ll share screenshots once I have some available.