BlueMix Hands-On Workshop - Meetupfiles.meetup.com/14755122/IBM BlueMix Workshop - Lab E - MapReduce...
Transcript of BlueMix Hands-On Workshop - Meetupfiles.meetup.com/14755122/IBM BlueMix Workshop - Lab E - MapReduce...
BlueMix HandsLab E - Using the BlueMix MapReduce Service to build an IBMBig SQL application
Version : 3.00Last modification date : 05/Owner : IBM
BlueMix Hands-On WorkshopUsing the BlueMix MapReduce Service to build an IBM
.00/11/2014
IBM Ecosystem Development
Using the BlueMix MapReduce Service to build an IBM
Copyright IBM Corporation 2013-2014. All rights reserved. 3
Table of Contents
Lab E: Using the BlueMix MapReduce Service to build an IBM Big SQL application .................................... 5
1. Creating a BlueMix MapReduce Service instance............................................................................ 5
2. Uploading the social media data to the BlueMix MapReduce Service instance............................... 5
3. Importing the data into BigSheets ..................................................................................................... 7
4. Exporting the workbook data............................................................................................................. 8
5. Launch Eclipse to bring up the sample Big SQL application ........................................................... 9
6. Define the BlueMix Server ..............................................................................................................10
7. Push the BigInsights sample application from Eclipse ...................................................................11
8. Running the application...................................................................................................................13
9. Validating the data import using the BigSQL console.....................................................................16
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 5
Lab E: Using the BlueMix MapReduce Service to build an IBMBig SQL application
Lab Objectives: This lab will guide you to understand and use the sample application that is provided for the
BlueMix MapReduce Service. You will learn how to work with the BlueMix MapReduce cloud offering.
As a Hadoop developer, you can get started quickly by using the BlueMix MapReduce Service (that is powered by
IBM® InfoSphere® BigInsights™) to create big data applications.
There are many advantages of developing this type of application on BlueMix.
• You can avoid the overhead of acquiring and provisioning your own hardware cluster.
• You can create an application by combining multiple BlueMix services
• You can scale yourhardware environment as your application needs increase.
You will use InfoSphere BigInsights BigSheets component that comes with BlueMix Map reduce Service to learn how
to explore social media data. This data is a look at the global coverage of the “IBM Watson” brand. Then, you will
learn how to query the data by using IBM Big SQL, which is the InfoSphere BigInsights SQL interface. You will then
deploy a BlueMix Java application that reads the data. The result of this exercise is an application that you
create for your users to dynamically run queries on that data.
Lab Duration : 30 minutes
1. Creating a BlueMix MapReduce Service instance
In this section you'll create the instance of the BlueMix Map Reduce Service that will be used throughoutteh rest of this lab.
Step 1 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary
Step 2 Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of thepage to take you there)
Step 3 Scroll down to the Services section and click on Add a service
Step 4 Scroll down to the Big Data category and click on the icon for the MapReduce service
Step 5 Click on CREATE AND ADD TO APP
Step 6 For Add to: select [Do Not Associate] in the pulldown. Leave the default name.
Step 7 Click Create
Step 8 Your new service should now appear in the Services section
2. Uploading the social media data to the BlueMix MapReduce Serviceinstance
In this section you'll upload social media data about Watson to the instance of the BlueMix Map ReduceService that you just created
Step 1 In the Services section of your BlueMix Dashboard click on the hexagon for the service youjust created to see more information about it
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 6
Figure 1 MapReduce service icon
Step 2 You see the URL and the login information that you need to open the services underlyingInfoSphere BigInsights Console.
Step 3 Write down the Username assigned to this instance (note: it is case sensitive). You'll need itlater on in the lab.
Step 4 Click on Launch the Console to bring up the console in another tab
Figure 2 Launching the console
Step 5 From the console click the Files tab
Step 6 In the DFS navigator, expand the user directory and select the directory with the same username that was in the service details
Step 7 Click the Upload icon.
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 7
Figure 3 Upload icon
Step 8 Click Browse and select the file \BlueMixLabs\LabE\blogs-data.txt where \BlueMixLabs isthe root folder of the files provided to you by the instructor. Click OK.
3. Importing the data into BigSheets
Now that you have the sample data uploaded to HDFS, you can import the data into BigSheets andcreate a workbook that contains that data.
Step 1 From the console click the BigSheets tab and click New workbook.
Figure 4 Create BigSheets Workbook
Step 2 Name the workbook WatsonBlogData. In the distributed file system browser, select the fileblogs-data.txt that you uploaded in the previous step.
Step 3 In the preview pane on the right , select a new reader to map the data into a spreadsheetformat. (Currently the reader is Line Reader.) Click the edit icon next to 'Line Reader' andselect JSON Array from the drop-down list.
Figure 5 Changing the reader
Step 4 Click the green check mark to change the reader. Click Fit Columns in the preview pane to
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 8
make the tabular data appear more compact.
Figure 6 Fit Columns to width
Step 5 Click the green check at the bottom to save the workbook.
Step 6 From the BigSheets workbook page, click Build new workbook to create a workbook with thesame data.
Step 7 Because you do not need all of the columns to complete your design, drop some of thecolumns. Click the arrow in any column heading, and select Organize columns
Step 8 Click the x next to the following columns to remove them:
▪ Crawled
▪ Inserted
▪ IsAdult
▪ PostSize
Click the check at the bottom to confirm removal.
Step 9 Click Save. Then, click Save and Exit,.
Step 10 Change the name of the workbook to WatsonBlogsDataV2 and click Save.
Step 11 Click Run to run the workbook. Wait for the run to complete.
4. Exporting the workbook data
In this section you’ll export the data from your updated workbook so that it can be loaded into BigSQLlater and then accessed using standard SQL queries
Step 1 In the menu bar of the WatsonBlogDataV2 workbook, click Export data.
Figure 7 Export Data menu item
Step 2 In the drop-down window, select the TSV type in the Format Type field
Step 3 Specify Export to File.
Step 4 Click Browse to select a destination directory in the HDFS file system. Select your path/user/<username and then type the name for the new file, WatsonBlogs Click OK .
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 9
Figure 8 Naming the exported file
Step 5 Make sure that the Include Headers check box is cleared, because you want to export thedata only. Click OK. A message dialog shows that the workbook is successfully exported , ClickOK.
Figure 9 Don't include headers
5. Launch Eclipse to bring up the sample Big SQL application
In this section you’ll start Eclipse and load the workspace that contains the sample Big SQL applicationyou'll use to interact with your BlueMix MapReduce service
Step 1 Open Windows Explorer and launch the program \BlueMixLabs\eclipse\eclipse.exe where\BlueMixLabs is the root folder of the files provided to you by the instructor.
Step 2 Select the workspace \BlueMixLabs\workspaces\LabE when prompted and click OK
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 10
Step 3 The sample BigInsightsSample should appear in the Project Explorer
Figure 10 BigInsightsSample project in Eclipse
Step 4 Feel free to explore the code . BigSQLJdbcClient.java is the interface to BigSQL.BigInsightsServlet.java handles all the URL requests
6. Define the BlueMix Server
In this section you'll define the BlueMix server so you can automatically push your Eclipse code out toBlueMix when needed
Step 1 Click on the Servers tab at the bottom of the Eclipse UI as shown in Figure 24.
Figure 11 Eclipse Servers tab
Step 2 Click the link labeled ...Click this link to create a new server.
Step 3 Expand the IBM category and select IBM BlueMix. Click Next.
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 11
Figure 12 Select IBM BlueMix as server type
Step 4 In the next dialog enter your BlueMix credentials and click on Validate Account
Figure 13 Validate your BlueMix account
Step 5 Verify that the dialog reports that your account is valid and then click Finish.
Step 6 Your server definition should now appear in the Servers tab
Figure 14 Your server definition
7. Push the BigInsights sample application from Eclipse
In this section push the sample appication to BlueMix, bind it to your existing MapReduce service andstart it from within Eclipse .
Step 1 In Eclipse go to the Servers tab and select your server, right click and select Add andRemove from the context menu
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 12
Figure 15 Add project to server
Step 2 Select the BigInsightsSample application and click Add .
Figure 16 Add project to BlueMix
Step 3 Click Finish. Click Next in the resulting dialog
Step 4 Append a unique string to the Subdomain (eg your first initial + last name ) since this has to beunique systemwide. See Figure 17 for an example. Click Next
Figure 17 Specify unique Subdomain for app
Step 5 Select your previously create MapReduce service. Click Next
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 13
Figure 18 Add existing service
Step 6 Click Finish to push and start your application.
Step 7 Messages should start appearing on the console. Wait for the message that says The serverdefaultServer is ready to run a smarter planet.
Figure 19 Console messages
8. Running the application
Now that the application has been deployed and started, you can run it from inside Eclipse
Step 1 From the Eclipse menu select Window>Web Browser and then select one of your installedWeb browsers (eg Firefox)
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 14
Figure 20 Select an external browser
Step 2 In the Project Explorer select your project, right click and then select Run As->Run onServer from the context menu.
Figure 21 Launch application
Step 3 Click on Finish in the resulting dialog. .The browser you selected in Step 1 should launch withthe application
Step 4 The application will load the file you exported in BigSheets into BigSQL. You have to providethe name of the file and the name of the new table in BigSQL that will host it's contents. Type in
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 15
WatsonData.tsv as the TSV File and my_demotable as the BigSQL table name
Figure 22 Exported Data File and BigSQL Table Name
Step 5 Clock Load. Verify that the returned message indicates a successful load. Note: if it failscontact your instructor
Figure 23 Successful Load
Step 6 Click Fetch data and verify that the first 5 rows of data are returned from BigSQL
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 16
Figure 24 Successful fetch
9. Validating the data import using the BigSQL console
In this section you’ll the data import performed by the application by looking directly at the BigSQLconsole
Step 1 In your browser go to the BlueMix URL https://ace.ng.bluemex.net and login if necessary
Step 2 Make sure you're in the Dashboard tab (if not click on the Dashboard link at the top of thepage to take you there).
Step 3 In the Services section of your BlueMix Dashboard click on the hexagon for the Map Reduceservice you've been using in this lab
Step 4 You see the URL and the login information that you need to open the services underlyingInfoSphere BigInsights Console.
Step 5 Click on Launch the Console to bring up the console in another tab
Step 6 In the console on the right side of the screen under Quick Links click on Run Big SQLQueries
Figure 25 Launch Big SQL console
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 17
Step 7 You'll need your BigSQL database name to proceed.Go to tab in your browser where you haveyour BlueMix environment running.
Step 8 Go to the Dashboard area and click on the square icon for you BigInsightsSample application
Step 9 Under App Runtime click on the round icon for the Liberty Runtime.
Step 10 Scroll down to Environment Variables and note the BigSQL database name as shown inFigure 26.
Figure 26 Big SQL database name
Step 11 Go back to the Big SQL console and enter the following command (replacing [db name] withthe actual database name you noted in the previous step.
Step 12 use [db_name]
Step 13 Click Run and verify that you get a successful status message.
Figure 27 Status message after use command
Step 14 Replace the use command with the following command and click Run again
select * from my_demotable
Step 15 Verify that the data you loaded via the applications is returned
Lab E: Build a BI Sample Application using MapReduce Service (Big Data) in BlueMix
Copyright IBM Corporation 2013-2014. All rights reserved. 18
Figure 28 Query results
Congratulations! You’ve successfully completed Lab E where you explored the BlueMix MapReduceservice that is powered by IBM BigInsights