BizTalk Server 2016 Feature Pack 1: Send tracking data to Azure Application Insights

Featured

Tags

,

BizTalk Server 2016 Feature Pack 1

Send tracking data to Azure Application Insights


You have configured various tracking options for orchestrations, send ports, receive ports, and pipelines using the BizTalk Server Administration console. And viewed tracked message events, message properties, and message bodies at various stages in a message’s flow in Group Hub pages in BizTalk Server Administration console. This data can be used for troubleshooting or auditing purposes.

The above discussed tracking feature is available in all BizTalk versions, including BizTalk 2016.

Feature Pack 1 goes a step further and provides feature to push the tracking data to Azure Application Insight. This enables non-technical users to troubleshoot and audit BizTalk Applications Tracked information can be monitored on Azure portal or with Power BI tools.

Follow the steps below to send tracking information to Azure Application Insight and then monitor that on Azure Portal:

Install BizTalk Server 2016 Feature Pack 1 on top of BizTalk Server 2016

Download BizTalk Server 2016 Feature Pack 1 from following link:

https://www.microsoft.com/en-us/download/details.aspx?id=55100

And, install BizTalk Server 2016 Feature Pack 1 on the server where you have BizTalk Server 2016.

Create Application Insight on Azure Portal

  1. Create a new instance of “Application Insight” in Azure Portal
  2. Navigate to the application insight you created in above step, Click Properties under Configure section and copy “INSTRUMENTATION KEY” from the blade on right hand side. This Instrumentation key will be required to configure Analytics in BizTalk environment, in next step.

1

Configure Analytics for BizTalk Environment

Open the BizTalk Server Administration console, expand BizTalk Administration, right-click the BizTalk Group, and select Settings. You notice that Red section, added by feature pack 1 for enabling Analytics.

2

  1. Check Enable group-level analytics.
  2. For the Target type, select Application Insight from the list.
  3. For the Connection parameters, enter your Application Insights instrumentation key copied earlier.
  4. Click “OK” to save your settings.

Now, BizTalk Server is ready to transmit data to Application Insights.

Enable Analytics on artifacts

Analytics can be enabled for receive ports, send ports and orchestrations, same like tracking.

Here in this scenario, I am enabling it for receive and send ports.

I have a pure messaging scenario, without orchestration, that receives a text file from one file location and drops at another file location. PassThru pipelines have been used both sides, and filter has been set on send port to subscribe messages from receive port, BTS.ReceivePortName == rcvPort

3

4

Follow the steps below to enable analytics on these 2 receive and send ports:

  1. In BizTalk Server Administration, right-click a receive port and select Tracking.

5

Notice that selection in Red. This was added by Feature Pack1

  1. Under Analytics, check Enable Analytics. And click OK to save settings.

This setting starts tracking and transmitting data from the artifact to Azure Application Insights.

Repeat above steps for send port as well.

Run the application once, by dropping sample text file.

Check Application Insight on Azure Portal

  1. Go to Azure portal and navigate to Application insight you created earlier.
  2. Click on Usage, and check Custom Events section on right blade (Red section below)

6

I dropped the file once, and I see 4 events altogether send to Application Insight.

These 4 events are nothing but what you see in Tracked Message Events in Group Hub in BizTalk Admin Console:

7

If you click one event, on Azure Portal, you will see all event data:

8

Right most blade in above diagram shows event data, BizTalk 2016 pushed to Application Insight.

Let’s compare Custom Data, shown in right most blade above with what has been tracked in BizTalk server:

CustomContextData

It’s almost same. BizTalk Server pushes all context properties as Custom Data to Azure Application Insight.

For more details, you can go to Analytics section directly, as shown below:

10

11

Click on USAGE

12 

This is all BI stuff now. You can change chart type here, select dimensions and measures etc.

You can check data in Tabular form as well:

13

Also, you can modify query on top window and execute it (select the query and click GO on top right)

14

And drill down to details of an event:

15

You can export Power BI query, as shown below, and use that with Power BI Desktop tool:

16

BizTalk Azure for Beginners – Part2

In Part1 in the series BizTalk Azure for Beginners, we prepared the environment to start “BizTalk Services” development. In this part, we will do some actual development and build our first BizTalk Service application. And deploy on Azure Cloud. This BizTalk Service will move an order flat file from one FTP to another FTP location after transforming it into XML format.

Scenario

This is a simple scenario of receiving an order flat file from FTP location, transform into an xml order and send to another FTP location. Both the FTP locations are on Azure VM under one FTP site.

scenario_simple

Sample Order Flat File, (|) delimited:

BLOB|Customer1|213|5th Street|New York|NY|12345
02/01/2015|10
02/02/2015|20
02/03/2015|30

 Structure:

Header:        Product Name | Customer | Address1 | Address2 | City | State | Zip
Line Item:    Date | Quantity

And XML structure:

OrdersXML

Create a BizTalk Service on Azure Portal

This is the service where we are going to deploy our BizTalk Services application.
Create a BizTalk Service on your Azure Portal:

btservice

You may require to download SSL certificate on your development server, from Azure Portal from the Dashboard of above BizTalk Service:

Install the certificate on in “Personal” and “Trusted Root Certification Authority”.

This certificates authenticates VS on your VM to connect to “BizTalk Service” on cloud so that you can successfully deploy BizTalk Service applications.

btservice_downloadsslcertificate_Install

btservice_downloadsslcertificate_Install1

Create FTP site on Azure VM

We need to create FTP receive and send locations for our scenario.
Create FTP Site on your Azure VM with Basic Authentication:

FTPSite

 

Which points to FTPFolders in Windows Explorer:

 

FTPSite_Folders

Open data channel port range (1035-1040) for FTP site for passive mode:

FTPFireWallSupport

Open the range of ports (1035-1040 and 21) on Azure Portal for your VM.
On Azure Portal, select your VM, go to “ENDPOINTS”, and click ADD at the bottom:

FTPOpenRangeinAzureforVM

FTPOpenPorts

Stop and Start FTP Service on VM using net stop ftpsvc and net start ftpsvc on command prompt.

 

Create BizTalk Service Project in Visual Studio

Create a project in visual studio of type “BizTalk Service”, as shown here:

btsproject

Create a Flat File schema for the sample Order Flat File:

OrderFFschema

Create an xml order schema:

OrderXMLSchema

Create a Map between OrderFF and OrderXML:

map

Open MessageFlowItinerary.bcs
This is the design surface where we need to put our work flow:

designsurface

You see Source, Destinations and Bridges in Toolbox.
Drag a Source FTP and Destination FTP and configure them:

designsurface_FTP

Server Address is Public IP of your VM on Azure which you can find from Azure Portal. Remember, this IP will change once you deallocate and reallocate the VM on Azure Portal. So, every time you do that you need to provide the new IP here on BizTalk Service project and redeploy on Azure.

Drop an XML one-way bridge in the middle. Double click the bridge and configure:

designsurface_bridge

Save and Close this BridgeConfig. Pick up connector from tool box and connect Source FTP to Bridge and Bridge to Destination FTP.

BTSExplorer_BridgeDetails

Select Bridge to Destination connector and set Filter condition and route action:

designsurface_FilterCond

Right click project name and say Deploy, and provide BizTalk Service credentials:

designsurface_Deploy

Check the refresh box and Deploy.
Once deployed, you can see your BizTalk Service’s artifacts:

btservice_deployedartifacts

Drop a Orders Flat file at FTP receive location (C:\FTPFolders\OrdersFF in our case).

BLOB|Customer1|213|5th Street|New York|NY|12345
02/01/2015|10
02/02/2015|40
02/03/2015|30

 You will see processed, transformed Orders XML at C:\FTPFolders\OrdersXML folder.

OrdersXML

BizTalk Azure for Beginners – Part1

Tags

,

This first article in the series “BizTalk Azure for Beginners“, is going to list all the tools required before you start developing BizTalk Services on Azure platform. You will figure out what are those tools, why they are required and how to install/configure them.

Assumptions

  • You are subscribed to Windows Azure and have an azure account.
  • Already have on-premise machine or created one as Azure VM with following configuration:
    1. Windows Server 2012 Datacenter
    2. SQL Server 2012 Enterprise
    3. Visual Studio 2012 Professional or Ultimate
      Note: do not install VS 2013 as available BizTalk Services SDKs are not compatible with VS 2013/15)
    4. BizTalk Server 2013 Developer / Standard / Enterprise edition
      Note: Developer edition is least costly on Azure but not available through Azure subscription. Available only through MSDN subscription. So, download it on VM through your MSDN subscription
      Note: Do not use BizTalk Server 2013 R2 as that requires VS 2013; and BizTalk Services SDK is not compatible with VS 2013.

Tools

Tool Azure Storage explorer
Get It From http://azurestorageexplorer.codeplex.com/
Purpose To view data in Azure Storage, blobs, queues and tables. Also helps in creating these artifacts and in generating SAS signature

 

Tool Windows Azure SDK VS 2012
Get It From Download from http://azure.microsoft.com/en-in/downloads/ For VS2012 or directly from Visual Studio “Start Page”
Purpose Integrated to VS 2012.

  • In Server Explorer, instantly view information about running cloud services, storage accounts, virtual machines, and service bus resources.
  • Also in Server Explorer, filter your view of resources by subscription, region, or both to show only the information that you care about right now.
  • Manage Windows Azure Virtual Machines from Server Explorer. Also, start a remote desktop (RDP) session directly from a virtual machine or a cloud service in the Server Explorer window of Visual Studio.
  • Create tables in Windows Azure storage, and edit table data.
  • Easily work with storage account resources from Server Explorer. View, upload, and edit blobs in your storage accounts. Create Windows Azure queues, and send messages to them, all without writing any code.
  • Automatically download storage account information from your subscription.
  • Publish to a Windows Azure appliance directly from the Windows Azure Publish Wizard.
  • Explore workflow in worker roles by using service bus tasks and queues in Server Explorer. Also, create worker roles that use service bus queues.
  • Deploy cloud services more quickly
  • Command line tools for packaging

 

Tool Azure Management Studio- Cerebrata
Get It From http://www.cerebrata.com/products/azure-management-studio/introduction
Purpose I really required it just to generate a working SAS signature.Somehow, SAS signature generated by “Azure Storage Explorer” was not working for me.

 

Tool BizTalk Service Explorer
Get It From Visual Studio Gallery
Purpose The BizTalk Service explorer is a Server Explorer Plugin for Windows Azure BizTalk Services. Use this to add your BizTalk Service to server explorer and be able to monitor your BizTalk Service deployment. The capabilities of this tool are:
(1) Fetch all artifacts (schemas, transforms, certificates, assemblies).
(2) Fetch all bridges.
(3) Delete any artifact.
(4) Delete any bridge.
(5) Upload an artifact from a file.
(6) Download an artifact and save it to a file.
(7) Add nodes for Sources and Destinations.
(8) Capture tracking information of a bridge and display
(9) Allow test messages to be sent to a bridge.

 

Tool Service Bus Explorer
Get It From https://code.msdn.microsoft.com/windowsapps/Service-Bus-Explorer-f2abca5aWritten by “Paolo Salvatori”.
Purpose Helps in viewing messages in Service Bus queues and topics.The Service Bus Explorer allows users to connect to a Service Bus namespace and administer messaging entities in an easy manner. The tool provides advanced features like import/export functionality or the ability to test messaging entities and relay services.

 

Tool Windows Azure BizTalk Service SDK for VS 2012 (WABS)
Get It From http://msdn.microsoft.com/en-us/library/azure/hh689760.aspx
Purpose The Developer SDK is required to develop Windows Azure BizTalk Services applications using Microsoft Visual Studio. This feature installs the following components:

  1. Microsoft WCF LOB Adapter SDK
  2. Microsoft BizTalk Adapter Pack
  3. Windows Azure BizTalk Services SDK

The BizTalk Adapter Service Runtime is required to connect to an on-premise Line of Business Application from a Windows Azure BizTalk Services application.

The Windows Azure BizTalk Services Tools installs the PowerShell extensions for managing Windows Azure BizTalk Service and BizTalk Adapter Service.



 

Windows Azure SDK VS 2012

Before you install the SDK, you have following view of VS:

1

You do not have Azure Explorer (like the one below) in “Server Explorer” and when you try to create a new project and select “cloud”, you will see above screen.

2

Install Windows Azure SDK for VS 2012 From Visual Studio “Start Page”-

3

Click on “Get the Windows Azure SDK”

4

Click on “Install the SDK for VS 2012”.

It will download VWDOrVs11AzurePack (Web Platform installer package).

This Installer can be downloaded from azure downloads page as well: http://azure.microsoft.com/en-us/downloads/

5

And click “VS 2012 install” and It will download VWDOrVs11AzurePack (Web Platform installer package)

Run the Web Platform installer package.

6 7 8 9 10

Now, you can see the difference:

11

We have Azure Explorer in Server Explorer and we have a new Project Template (Azure Cloud Service).

Azure Storage Explorer

Download Azure Storage explorer from the following link, unzip and install

http://azurestorageexplorer.codeplex.com/releases/view/125870

12

Open Azure Storage explorer and Add your Azure Account:

13

Here you need to provide your storage account information which you can get from Azure Portal: focus on storage account and click “Manage Access Key”:

14

Now, you can see blobs, queues and tables in azure storage:

15

 

Azure Management Studio from Cerebrata

One tool to manage your Microsoft Azure cloud storage, diagnostics data and application workflows.
I really did not need it as I already have Windows Azure Explorer and Azure Storage explorer, but somehow, SAS signature generated by this tool only, worked for me.
So get Trail copy from: http://www.cerebrata.com/products/azure-management-studio/introduction and install.

Open Azure Management Studio:

16

Click Add Storage Account Connection and provide the same information you did for Azure Storage Explorer and then you can view your Azure Storage in Azure Management Studio:

17

 

Windows Azure BizTalk Services SDK for VS 2012

Download the SDK (Microsoft Azure BizTalk Services SDK Setup) from http://msdn.microsoft.com/en-us/library/azure/hh689760.aspx

Unzip the downloaded file, it will automatically run installation:

18

Select all three and say Next.

It will ask the name for “BizTalk Service” (to connect to) as well on your Azure portal.
Provide name of “BizTalk Service” you created on your Azure Portal or leave it for now.
Provide any name, it will not connect right now. We can do it in next part in this series.

It will install Web Service in IIS:

19

After successful installation, you will notice BizTalk Adapter Services in Server explorer:

20

 

Biztalk Service Explorer

The BizTalk Service explorer is a Server Explorer Plugin for Windows Azure BizTalk Services. Use this to add your BizTalk Service to server explorer and be able to monitor your BizTalk Service deployment

Get it from Visual Studio Gallery:

BTSExplorer_VSGallery

Download and run the MSI.

BTSExplorer_AddBTService

Now you can see the details of the BizTalk Service artifacts like Bridge, schema, maps etc. and also tracking information.

BTSExplorer_BridgeDetails

 

Service Bus Explorer

The Service Bus Explorer allows users to connect to a Service Bus namespace and administer messaging entities in an easy manner. The tool provides advanced features like import/export functionality or the ability to test messaging entities and relay services.

Download the zip file from https://code.msdn.microsoft.com/windowsapps/Service-Bus-Explorer-f2abca5a, and unzip. Run the application from C#\bin\debug location, Provide connection string and connect to your Azure Service Bus:

ServiceBusExplorer

 

That’s all. You are ready to create your “BizTalk Services” Projects.

BizTalk Azure for Beginners

Tags

,

This Series of posts is for beginners who want to do BizTalk development on Microsoft Azure Platform.

Part1

The part will detail the steps for setting up environment for BizTalk Services development and, second part takes through actual development process.

https://biztalkjunkies.wordpress.com/2015/01/26/biztalk-azure-for-beginners-part1/

part2

In this part, you will actually develop “BizTalk Services“, and deploy them on Azure Cloud. It’s a simple scenario of receiving an order in flat file format from a FTP location, transform to XML order and send to another FTP Location.

https://biztalkjunkies.wordpress.com/2015/02/02/biztalk-azure-for-beginners-part2/

PART3

This is a complex scenario where you will post messages to Azure Storage Blob, Service Bus Queue and Service Bus Topic.

Coming Soon…..

Integrate 2014 – Microsoft Campus, Redmond, WA – December 3-5, 2014

Day03_08_Andrew_Slivker_SOA_Governance_for_BizTalk_Server Day03_05_Paul_Larsen_Byram_Informix Day01_06_Guru_Venkataraman_BizTalk_Server_Roadmap Day02_01_Anurag_Dalmia_Business_Rules Day01_05_Business_To_Business_Harish_Kumar_Agarwal Day03_03_Michael_Stephenson_Hybrid_Connevtivity_Options Day03_02_Kent_Weare_How_To_Choose_An_Integration_Platform Day03_01_KannanIyer_Investing_in_Cloud_integration Day01_03_Alok_Jain_Business_Process_Management Day01_04_Sameer_Chabungbam_Enterprise_Application_Integration Day01_02_Bill_Staples_Application Platform & Integration Platform Strategy and Roadmap Day02_06_Alan_Scott_Clinical_Innovation_BizTalk Day02_05_De_Jonge_Power_BI Day01_01_Scott_Guthrie_The_Cloud_for_Modern_Business Day02_04_Josh_Twist_Azure_API_Management Day02_03_WeeHyong Tok_Data Factory_Integration