Understanding RosettaNet Message Flow in BizTalk Server 2006

If a picture is worth a thousand words, then these two diagrams (courtesy & copyright Microsoft) are worth millions! At least they are to anyone desperate to understand how the BizTalk Accelerator for RosettaNet (BTARN 3.5) really works.

Initiator Message Flow

Don’t let the complexity of this diagram scare you. The message flow for an outbound RNIF (or PIDX) message consists of four key areas.

Area 1: The SQL Server BTARN Databases where all “lob” messages (such as a P21 – PIDX Invoice) and any attachments are stored BEFORE the outbound RNIF message flow actually begins, and the SQL Receive Location used to process the message content and attachments into the MessageBox. 

Note: If you’re not sending attachments you may elect not to use SQL for this and can substitute FILE receive locations. The SDK contains sample code and a pretty good explanation of how this may work.

Area 2: The Initiator Private Process orchestration where the actual messages and attachments are “prepared” (usually transformed into XML) for further processing by the initiator public process.

Area 3: The Initiator Public Process orchestration where the outbound message is actually formatted to meet the RNIF standard. This includes adding the required RNIF headers and other such tasks. Consider this a “Black Box” since you can’t change how this works without invalidating the RosettaNet certification.

Area 4: The HTTP/HTTPS Send Ports and the RNIFSend.aspx web application that are CRITICAL to the correct processing of the outbound messages. This is where the real work is done to validate the messages, create the correct MIME headers and add the required RNIF DOCTYPE declarations.

Just a side note. If you’re really interested in how this all works, you can look at the source code for this web application using Lutz Roeder’s Reflector. I think you’ll be amazed at how much Regex is used to “craft” these RNIF messages.

Responder Message Flow

The message flow for an inbound RNIF message (whether it’s a new message or an asynchronous response to a message you’ve already sent) consists of three key areas.

Area 1: The RNIFReceive.aspx web application which processes the inbound message, validates it and then removes all the RNIF specific headers and DOCTYPE declarations. This then passes the message along to the HTTP/HTTPS Receive Port and Pipeline which handles the non-repudiation, decodes the message, parses the message and resolves which Party it came from.

Area 2: The Responder Public Process receive the RNIF message from the MessageBox and extracts the content and attachments and sends them to the responder private process for further processing. Just like the initiator public process, the Responder Public Process should be considered a “Black Box” and left unchanged.

Area 3: The Responder Private Process then routes the inbound messages and any attachments to the SQL Server BTARN Databases where it can be further processed and sent to some “lob” back-end system.

There are a few other details involved in the message flow (signals) depending upon whether your transmission is synchronous or asynchronous, but the basics are fairly straight-forward. The best thing is that this workflow is almost entirely “configured” using the BTARN Management Console and is “message agnostic”, meaning that you can use any standard RNIF, CIDX or PIDX “PIP” (Partner Interface Process) that you and your trading partner both support.

Best of all, it’s included in the price of BizTalk Server 2006 R2!

Currently listening to Carrie Underwood’s Carnival Ride

PIDX Schemas Solution for BizTalk Server 2006

One of the things you’ll need when working with PIDX / RNIF / BTARN projects in BizTalk Server 2006 is the actual PIDX schemas (.xsd) to be used. Luckily the American Petroleum Institute (API) has a public web site for the Petroleum Industry Data Exchange (PIDX) committee which contains links to the various versions of their schemas.

Once you and your trading partner decide on which version you’ll be using, you can download the schemas and add them to your BizTalk solution in Visual Studio. When you attempt to open these schemas you’ll notice an error message that pops up as shown here.

This is a fairly common error and usually occurs when the schema you are attempting to open contains a reference to another “imported” or “included” schema and the BizTalk Schema Editor tool cannot find the file in question. The work-around for this is fairly simple and all you’ll need to do is change these two attributes.

Step 1: First you’ll need to click on the Imports attribute collection and delete the reference to the PIDX Library schema which contains the hard-coded location which is incorrect.

Step 2: Next you’ll need to import a new schema as an XSD Include and select the PIDXLib from the picker as shown here.

Step 3: Once that step is complete, you’ll also need to change the Root Reference attribute so that the BizTalk Schema Editor knows how to correctly identify your schema.

And that’s really all there is to it, except you’ll have to do this for every PIDX schema and every version. To make this easier for everyone else using PIDX schemas in BizTalk Server, I’ve gone ahead and created a BizTalk Solution that includes all the PIDX schemas for their versions 1.0, 1.2 and 1.3 and you can download the zip file below.

Currently listening to Brad Paisley’s 5th Gear.

BizTalk Server 2006 - BTARN 3.5 Configuration Tips

Day 5 in my RosettaNet implementation nightmare and I can finally see the light at the end of the tunnel!

Just a quick post to those of you planning to use the BizTalk Accelerator for RosettaNet 3.5. There are a few little “issues” that you need to know about before you get started.

Accounts, Hosts & Host Instances

Hosts must have the Authentication Trusted option enabled. This has to be configured BEFORE you create any Host Instances. I recommend creating a new Service Account, a new In-Process Host, a new Isolated Host and then a new Host Instance for each of these as shown in the table below. You will also need to create new HTTP and SQL Send Handlers and Receive Handlers as well.

Note: The In-Process Host Instance account and Isolated Host Instance account must be the same for BTARN to work properly!

Service Account:	Domain\AccountName
In-Process Host:	RosettaNetApplication
In-Process Host Instance:	RosettaNetApplication
Isolated Host:	RosettaNetIsolatedHost
Isolated Host Instance:	RosettaNetIsolatedHost
SQL Send Handler:	RosettaNetApplication
SQL Receive Handler:	RosettaNetApplication
HTTP Receive Handler:	RosettaNetIsolatedHost
HTTP Send Handler:	RosettaNetApplication

Installation & Configuration

Don’t use “localhost” for your BizTalk Server or Web Server name when running the Configuration Wizard. Use the NetBIOS name of your BizTalk Server machine and the NetBIOS name of your Web Server machine instead. The Configuration Wizard will fail with a strange error if you use default “localhost”.

Why? I have no earthly idea but I will be submitting this to the BizTalk team as a bug!

One final note

All the BTARN bits are deployed to the BizTalk Application 1 by default. I’m still trying to find a way to isolate the RosettaNet bits (schemas, maps, pipelines, ports and orchestrations) in their own BizTalk Apllication much like the new EDI/AS2 bits are. I’ll post if I find an easy way to accomplish this.

Happy Integrations!

Update

I just want everyone to know that I’m not really all that smart. I had a ton of help setting up BTARN 3.5 and getting things working properly! I hired the folks from Microsoft Consulting Services to come in and teach me everything I could learn about using RosettaNet in BizTalk Server 2006. Special thanks to Ross Santee! A great consultant.

Currently listening to: Dave Koz – Castle of Dreams

Tips for Using PIDX Schemas in BizTalk Server 2006

Day 3 in my RosettaNet implementation nightmare and I still can’t see the end of the road!

As you may or may not know, the American Petroleum Institute (API) has it’s own set of XML schemas and transport standards know as the Petroleum Industry Data Exchange or PIDX for short. This is the actual “standard” that my current RosettaNet project is really all about. The interesting about these schemas is the fact that they use their own unique namespace and prefix as shown below:

xmlns:pidx=”http://www.api.org/pidXML/1.0“

Since the BizTalk Mapper tool always generates XML document with the “ns0″ prefix and there is no property you can set to change this, you’ll need to use a little “XSLT slight-of-hand” to get this to work.

Step 1: Create your BizTalk Map as you normally would using the mapper tool.

Step 2: Validate your map and open the xslt generated by BizTalk Server 2006 in your favorite text editor. This file can usually be found in the “\temp\_mapdata” folder.

Step 3: Replace all instances of the namespace prefix “ns0″ with the prefix you require (which in this case is “pidx”) as shown below.

Step 4: Save the result as an .xslt file and include it (or add it) into your BizTalk project.

Step 5: Create a new map using the same source and destination schemas you used in Step 1 but use the external xslt you generated by setting the “Custom XSLT Path” property on your new map.

The results are shown in the figure below. Viola! The “ns0″ namespace prefix has been replaced by the desired “pidx” prefix.

Click below for more information about using external xslt in your maps.

Custom XSLT in BizTalk Maps

Take Away: There is always more than one way to skin a cat! In BTS2006 it may not be obvious how to do something a little unusual.

Business Process Automation – Don’t Just Shift the Costs!

I’m stepping on my soapbox again so be warned!

Yes, I know I’ve said this before…

I’ve recently encountered a number of well-intentioned (or perhaps not so well-intentioned) companies looking to reduce their internal costs for processing orders and invoices through some sort of “business process automation” system. Being an avid BPA developer I applaud these efforts so long as they bring “value” to everyone in the supply chain. What I see all too often is companies spending thousands and thousands of dollars on web-based initiatives where the main goal is to shift the transaction processing from themselves to their suppliers. From a business perspective, this just isn’t a wise choice and from a technological perspective, it just isn’t necessary. Having your suppliers enter their invoices first on their ERP system and then on your web-based system does not reduce the “transaction cost” it just shifts the labor from yourself to your supplier.

True B2B Integration Doesn’t Just Shift Costs, It Eliminates Them!

The Internet offers developers the unparalleled opportunity to connect disparate systems together without paying EDI “VAN” (Value Added Private Network) charges, telecom “WAN” (wide area network) charges or even leased-line charges. Basically, all the infrastructure costs are essentially free or paid for though corporate web access. All the developer (and business analyst) has to do is “connect-the-dots” using products like Microsoft’s BizTalk Server, Tibco’s iProcess Suite or Software AG’s WebMethods product to eliminate the real transaction costs. It’s never been easier or more affordable to do this and the benefits to the entire supply chain are enormous! Automating the transaction processing so that the document (purchase order or invoice) is “touched” by human hands only during it’s creation doesn’t just shift costs, it eliminates them!

Yes, I’ve heard all the arguments about how “costly” B2B integration can be and how smaller companies really can’t afford to participate but IT JUST ISN’T TRUE! Products like BizTalk Server 2006 make BPA truly affordable for companies of any size. I know of at least one very small company that used this product to integrate QuickBooks (their “ERP” system) to their much larger suppliers and customers. If they can do this without breaking the bank, anyone can!

Developer Call To Action: The next time your senior management talks about automating business processes with customers and suppliers, don’t just develop a BPA system that shifts the costs, work to eliminate them altogether!

RosettaNet: Who Invents This Stuff?

Captain Picard - “Admiral, we’ve engaged the Borg”

Day 2 in my RosettaNet development project and all I can say is “Who Invents This Stuff”?

Yes, it’s been a glorious two days of trying to comprehend the world’s most complicated business-to-business process. Only the electronics industry could conceive of something so overly complex. Secure, internet-based transactions were never meant to be this difficult. It’s like swatting a fly with an atom bomb! After ten years of B2B development I thought I had seen it all. Boy, was I wrong!

And the worse thing is that you have to pay for the privilege of enduring this torture. Every other xml standards organization I’ve ever dealt with freely distributes their schemas and specifications for everyone to use. Not so for the folks at RosettaNet.org! You’ll have to pay anywhere from several hundred to several thousand dollars for the rights to use their “PIP”s (Partner Interface Process) which is really nothing for than an XML DTD or XSD (schema) and a process guide explaining the message flow. It looks like RosettaNet.org used to publish these on their web site but now you’ll have to pay for a subscription to get these.

Why on earth would so many smart B2B developers adopt such an overly complex, expensive and time consuming standard is beyond my understanding. Folks, business-to-business transactions do NOT have to be this difficult.

Stay tuned for more fun & games!

BizTalk Server 2006 R2: To Boldly Go Where No Man Has Gone Before!

This week I’m going to embark on a dark and mysterious journey, fraught with great danger, both real and imagined. I’m about to begin development of a BizTalk Accelerator for RosettaNet (BTARN 3.5) project with the assistance of Microsoft Consulting Services. For those of you experienced in BizTalk development, you’ll understand my trepidation. For those of you familiar with RosettaNet (RNIF), you’ll understand my sheer terror!

If I survive this ordeal, I promise to post about my experiences, so that future generations of BizTalk developers don’t make the same mistakes, don’t fall into the same traps and don’t lose what little is left of their rapidly graying hair!

Stay tuned!

Finding Freelance Work

I received a recent comment from Paul, a reader on my “other blog” asking how I found my “weekend gigs” or freelance development work. That’s a pretty good question and the short answer (without sounding too presumptuous) is that God provides! The long answer is “I don’t really know, it just seems to happen”.

I started my career many years ago as a degreed mechanical engineer, fresh from college and looking for fame and fortune in the “oil business” (which in Texas is correctly pronounced “Awl Bidness”). Unfortunately, I arrived on the scene just in time to watch oil drop from $40 per barrel down to $7 per barrel which left myself and about 100,000 other engineers scrambling for any work we could find. (If you’re a history buff or just follow the price of oil, you should be able to place my age within +/- 2 years from this information) Luckily, I landed a real engineering job for a valve manufacturer in Houston. I worked for that company for 18 years and watched it grow from $50 million in revenue to over $40 billion as it was acquired and reacquired over the next ten years. When I began with the company we had 300 employees and when I left the “company” we had over 240,000 employees and our CEO and CFO had just been indicted for tax evasion and securities fraud among other things. Care to guess the name of that company?

I held a number of engineering, product management and sales & marketing positions in that 18 year period and finally got tired of all the politics and corporate ladder climbing. So I asked the IT Director (a good friend) if he could find a position for me somewhere in the IT programming or operations area so that I could explore my love of computers, software and e-commerce. My friend and new boss gave me the opportunity to learn, do, and learn by doing and we had a blast. We put together that company’s first web site, first e-commerce site and first B2B system using pre-release versions of Microsoft’s SQL Server, BizTalk Server and Commerce Server. Over the next two years, with the help of some great people at Microsoft (yes, the Blue Monster really does have some great people) we designed and built a world-class B2B e-commerce system for (you guessed it) Tyco. One that has transacted literally hundreds of millions of dollars in transactions and is still in use almost ten years later.

My boss and I left Tyco, formed our own B2B consulting firm and as we had hoped, got Tyco as a client. The first year was great. We had lots of projects, worked 70 hour weeks and made good money. The second year taught us the lesson that most consultants come to call “going from feast to famine”. We called it something else (mostly unprintable) but learned several valuable lessons from the experience. I learned that I’m not cut out to be a full-time consultant and for me, it’s tremendously important to see “the fruits of my labors”. Which is why I work for a great medium-size “private” company today and do my “freelance” work in the evenings and on the weekends “as my time and energy permits”.

How do I find the work? I don’t really. It just seems to find me somehow, but I can give you a few tips to get started!

• Do volunteer work! It’s good for the soul and opens you to all sorts of opportunities.
   
• Give back to the community! Share your best work, start a blog or two. Post in the community forums.
   
• Answer your email and every (non-spam) blog comment! It’s amazing how word gets around the Internet.
   
• Try something new. Life is way too short to always take the safe road. Learn a new programming language. Hell, learn a new language period.
   
• Be courageous! Buy a Mac. Become a fanboy! Put an Apple sticker on your car.

Currently listening to Diana Krall’s “The Look of Love”.

More Thoughts on HTML5, CSS3 & WebKit Advances!

I thought I’d take this opportunity to explain a little bit more about why I believe the new “features” in WebKit are so important to the future of web development and design.

First off, let me set the record straight and tell you that I am an unqualified supporter of Microsoft technologies such as SQL Server, BizTalk Server, Commerce Server and the .NET Framework. I’m also a Microsoft MVP for Commerce Server and an avid C#, ASP.NET and BizTalk developer. In my day job, I use these Microsoft technologies to create business-to-business e-commerce applications for the company that I work for.

But at night and on the weekends, I moonlight as a freelance web developer using mostly non-Microsoft technologies such as Ajax, PHP & mySQL. In both areas I strive to create “standards” based web sites and applications and my overriding goal is always to “create the best user experience requiring the least bandwidth” and this is where WebKit comes in.

When you think about how ASP.NET developers create great user experiences today, two things come to mind; ASP.NET AJAX and Silverlight. Both technologies allow developers to create really great user experiences on the web but only at the cost of bandwidth (download time, initial or otherwise). The same rule holds true for Flash and any Ajax library such as Prototype, script.aculo.us or jQuery (all of which are excellent Javascript frameworks).

Now think about the potential to create great user experiences using nothing more than the new HTML5 and CSS3 capabilities found in the latest WebKit builds. Gradients, shadows and rounded-corners without images, transforms and animation without Javascript, client-side data that goes way beyond cookies and support for highly compressible vector graphics (SVG). All in a fully “standards” based HTML/XHTML/CSS framework that (hopefully) renders the same in all browsers, both desktop and mobile.

Now we’re talking about actually having the tools to “create the best possible user experience requiring the least bandwidth”. This may be a pipe dream but it looks like the WebKit folks and I are drinking the same Kool-Aid at the moment!

Currently listening to: “Still Feels Good” by Rascal Flatts

Why Safari May Become the Browser of Choice!

If you’re a web designer, web developer or just someone that keeps up with the latest “Web 2.0″ technologies, you know that a lot of progress is being made by ALL the major browsers to become “standards compliant”. You also know that the Web Standards Project has created a number of “Acid” tests that help all the browser developers ensure that their browser works as “expected”. If you’re an experienced web designer or developer, you probably use several different browsers (IE, Firefox, Opera, Camino, Safari, etc.) to test your sites against everyday.

What you may not be aware of is some of the very “advanced” features Safari (WebKit actually) has in the works which may well change the way we think about developing Web 2.0 applications.

1. Web Fonts
2. Client-Side Database Storage
3. CSS3 Transforms
4. CSS3 Animation
5. SVG Support
6. CSS Gradients
7. CSS Box Shadow
8. And Many, Many More…

If you look at any one of these new features individually, they are very cool! If you look at integrating these new features together, you begin to see the potential for replacing today’s Javascript (Ajax) “eye-candy” with native browser rendering support!

And why you ask, is WebKit (and Safari) pushing these advanced features out the door so quickly?

Safari on iPhone!

Starts you thinking, doesn’t it!

Currently listening to: Eric Merienthal’s “Just Around the Corner”