SQL Server 2000 makes it simple to export your data as XML; but importing XML and working with XML files are a little trickier.

If you consult Books Online (BOL), you'll find several relevant entries, including OPENXML and OPENROWSET. All the examples supply the XML text as a declared variable, which is convenient for the documentation staff but not for the developer who wants to read an XML file and treat it accordingly. To make sense of this onion, perhaps it's best to approach it from the inside out.

OPENXML is a rowset function (i.e., one that returns a rowset), and it works similarly to the rowset functions OPENQUERY and OPENROWSET. Using OPENXML, you can perform JOINs to XML data without actually importing the data first. You could also use it in conjunction with INSERT. . . SELECT, UPDATE, and DELETE.

However, to use OPENXML, you must perform two tasks that aren't required with OPENQUERY and OPENROWSET. These tasks require two system stored procedures.

The first, sp_xml_preparedocument, reads the specified XML text and moves its contents into memory. Here's the syntax:

sp_xml_preparedocument @hdoc = OUTPUT,
[, @xmltext = ]
[, @xpath_namespaces =

The parameters are:

  • @hdoc: a handle (effectively a pointer) to a region of memory where some data of interest resides. Note that it is an output variable--after the procedure runs, the variable will contain the handle to the XML file's contents in memory. Be sure to save the result since you'll need it later.
  • @xmltext: the actual XML text you want to process.
  • @xml_namespaces: any namespace references that your XML requires in order to operate correctly. Note that any URLs here need to be delimited with angle brackets (<>).

Assuming that the parameters you passed are valid and the XML text is sound, your XML data will reside in memory. Now you call sp_xml_preparedocument, passing the variable containing your XML file, and then OPENXML, whose syntax is a little more involved:

OPENXML(idocint [in],rowpatternnvarchar[in],[flagsbyte[in]])
[WITH (SchemaDeclaration | TableName)]

Note: There isn't enough space in this tip to describe the parameters that OPENXML accepts. See BOL for more information; search for OPENXML in Transact-SQL Reference.

Now we have the back end in place. All that remains is importing an actual XML file into SQL for processing. (It's curious how all the BOL examples leave this crucial part unattended.)

(I have to acknowledge the invaluable assistance of my colleague Billy Pang, who helped me work through this problem. He inspired the code that follows, though I tailored it for the specific requirements. Thanks, Billy!)

The basic trick is, read the file as text, line by line. Then, concatenate all the resulting rows into a large VARCHAR variable. Finally, pass this variable into the foregoing code.

Here is the code to read the file and store its contents into a variable:

Now you have the entire contents of the file in the variable @FileContents. All you have to do is plug said variable into the @xmltext parameter to sp_xml_preparedocument, and then call OPENXML.

With this solution in hand, it's possible to perform all the magic that you might want. You can join XML documents to SQL tables without importing the data, and then use the result to INSERT, UPDATE, DELETE for whatever your please.

Related links

Comments

1

Rahul Khinvasara - 25/01/05

What if the size of xml file is 50mb ?

How much a varchar field can hold ?

» Report offensive content

2

Randy Koehn - 19/04/05

You guys totally rock!!!

This was the solution that I have hunted for some time... brilliant; only thing is every XML file that I am working with is greater than 8000 characters long. I can easily enough spread the file across several varchar(8000) variables; however, I can't find a reasonable means to concatinate them for use with the sp_xml_preparedocument command... any suggestions?

» Report offensive content

3

Kumar Narala - 10/10/06

Thank you very much for sharing such an important piece, which has been omitted in most of the sites

» Report offensive content

4

harry - 11/10/06

what if the file size is greater than what can be stored in @filecontents (i.e) greater than 8000 which can be stored using varchar?

» Report offensive content

5

Moe - 10/01/07

I had the same idea but knew there was an 8000 limit on varchar. I see others had the same question. Anyone have a solution?

» Report offensive content

6

Paul Manley - 17/04/07

The BOL isn't missing anything. The addition of reading in a text file isn't part of the OpenXML problem. Also, using xp_cmdshell for this is pretty flawed.

If anything you should be using DTS, COM, setup a linked server to a file, do a bulk insert first to get your data, or preferably pass it in from your application.

The Text data type allows more than 8000 characters. And that is what you should be using for this kind of data.

The only issue I see with this is there seems to be no way to edit the recordset. Being able to do an update on this data source would be supremely useful.

» Report offensive content

7

Simon - 08/05/07

Thanks, This is a life saver. I've wanted to be able to read the file in transact SQL to be able to do my XML data load in a single transaction.

Unfortunately using DTS Com and using the SQL Server XML Bulk insert makes this very difficult.

» Report offensive content

8

Deepak - 13/09/07

Hi Arthur,

really good to see ur code for reading an xml file and then using it in OPENXML method.

But I am facing a problem here. I used your code to fill a variable with the data of an xml file. It got filled with all the data successfully. But when I used this variable in sp_xml_preparedocument method it doesn't returned any result.

Now I tried the other way. I copied all the data of the xml file and hardcoded that data into a @doc variable, inspiite of using your code to read the xml file. Now OPENXML worked perfectly and it returned records as expected.

I am using element centric mapping...

Can u suggest... I am attaching the code here...

/*** CODE TO READ XML FILE AND USE OPENXML METHOD ***/
/**************************************************/

declare @idoc int
declare @doc varchar(1000)
DECLARE @FileName varchar(255)
DECLARE @ExecCmd VARCHAR(255)
DECLARE @y INT
DECLARE @x INT
DECLARE @FileContents VARCHAR(8000)

CREATE TABLE #tempXML(PK INT NOT NULL IDENTITY(1,1), ThisLine VARCHAR(255))

SET @FileName = 'C:\Stu.xml'
SET @ExecCmd = 'type ' + @FileName
SET @FileContents = ''

INSERT INTO #tempXML EXEC master.dbo.xp_cmdshell @ExecCmd
SELECT @y = count(*) from #tempXML

SET @x = 0
WHILE @x <> @y
BEGIN
SET @x = @x + 1
SELECT @FileContents = @FileContents + ThisLine + char(10) from #tempXML WHERE PK = @x

END

print @FileContents
DROP TABLE #tempXML

exec sp_xml_preparedocument @idoc output , @doc

select * from
OPENXML
(
@idoc , '/ROOT/student' , 2
)
WITH
(
id varchar(10),
name varchar(10)
)

exec sp_xml_removedocument @idoc


/*** XML FILE DATA **/
/**************************************************/
<?xml version ="1.0" ?>
<ROOT>
<student>
<id>1</id>
<name>Deepak</name>
</student>
<student>
<id>2</id>
<name>Mahinder</name>
</student>
<student>
<id>3</id>
<name>Gokul</name>
</student>
<student>
<id>4</id>
<name>Raj</name>
</student>
<student>
<id>5</id>
<name>Ashok</name>
</student>
</ROOT>'

» Report offensive content

9

Vinu Joshua - 20/09/07

Good thought... but is there anyway to overcome the 8000 char limit?

» Report offensive content

10

kiran826 - 15/10/07

Hi i have used above piece of code to read an XML file but i didn't get any output. could anyone provide me a solution.

I have an xml file generated by a dataset (ds.ReadXML()). i want insert all these rows into DB. Please suggest me in this regards.

» Report offensive content

11

Mani - 03/11/07

Kiran826,
Make sure your xml file is residing on the same server as your sql server. Since this query is executed from the sql, it will be looking for this file on that server.

» Report offensive content

12

Manisha - 26/01/08

I am readinf my xml file the you have described here. But when the xml data is saved as text it "n++" prefix is being added to it. Hence the next operation of sp_xml_preparedocument throws xml parser error. I have attched xml format here.
could you please suggest something?

<?xml version="1.0" encoding="utf-8"?>
<Searches>
<Search ID="1" SearchCondition="" ElementID="14" SearchCriteria="equals" SearchValue="1/3/2008" />
<Search SearchCondition="OR" SearchValue="" SearchCriteria="Yes" ID="2" ElementID="6" />
<Search SearchCondition="OR" SearchValue="HTML" ID="3" SearchCriteria="equals" ElementID="1" />
</Searches>

» Report offensive content

13

Jitu Mevada - 14/05/08

I want to read below XML file into SQL...... I dont know How to get fields value. Please provide me proper help.

<?xml version="1.0" ?> 
- <Report xmlns="Agent_x0020_State_x0020_Statistics" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="Agent_x0020_State_x0020_Statistics http://report.tataindicomracc.com/ReportServer?%2fEtech%2fAgent+Reports%2fAgent+State+Statistics&rs%3aCommand=Render&rs%3aFormat=XML&rs%3aSessionID=4m1nto33jt2o0njp1tfw3pfb&rc%3aSchema=True" Name="Agent State Statistics">
- <Result>
- <Result_Group_Agent_Name_Collection>
- <Result_Group_Agent_Name textbox69="Abhishek Khandelwal">
- <Result_Group_Date_Collection>
- <Result_Group_Date textbox67="5/9/2008" TotalLoginTime="07:45:37">
- <Result_Group_Hour_Collection>
- <Result_Group_Hour textbox62="12:00 AM - 12:00 AM">
- <Detail_Collection>
<Detail state="Available" Duration="02:03:08" Average="00:00:27" textbox53="26.4452160217633" />
<Detail state="In Call" Duration="03:07:32" Average="00:00:40" textbox53="40.2763360418084" />
<Detail state="Released" Duration="01:52:38" Average="00:02:26" textbox53="24.1901421054516" />
<Detail state="Ringing" Duration="00:03:17" Average="00:00:01" textbox53="0.705158034148262" />
<Detail state="Wrap Up" Duration="00:39:02" Average="00:00:08" textbox53="8.38314779682858" />
</Detail_Collection>
</Result_Group_Hour>
</Result_Group_Hour_Collection>
</Result_Group_Date>
</Result_Group_Date_Collection>
</Result_Group_Agent_Name>
</Result_Group_Agent_Name_Collection>
</Result>
</Report>

» Report offensive content

14

Adam - 15/05/08

This is really a brilliant piece of work. I can't tell you how many times I get this sort of question. If I may make one suggestion, a lot of XML files have a newline character at the end, which might get convereted to a NULL when you import the results into a table. NULL plus anything is NULL. Pursuant, you might want to do this as you build the string:

Great work! Thank you for sharing!!

SELECT @FileContents = @FileContents + IsNull(ThisLine, '') FROM #tempXML WHERE PK = @x

» Report offensive content

15

Santosh - 03/07/08

I want to insert records from XML string. Means my sql parameter will be XML data and in store procedure i want to manupulate each record and store it into table.
Can anybody suggest me any solution?

» Report offensive content

16

Tashfeen Khalid - 10/02/09

Reading a file into SQL Server... http://whereclause.com/blog/?p=83 ...
Consuming XML in a stored procedure... http://whereclause.com/blog/?p=8 ...

» Report offensive content

17

Jon - 13/01/10

Excellent article. I really would like to see an example of how to do an update on the FileContents object and write it back out and overwrite the original XML doc on the file system. I'll have to figure it out myself otherwise, which will take forever.

» Report offensive content

Leave a comment

You must read and type the 6 chars within 0..9 and A..F

* indicates mandatory fields.

17

Jon - 13/01/10

Excellent article. I really would like to see an example of how to do an update on the FileContents object ... more

16

Tashfeen Khalid - 02/10/09

Reading a file into SQL Server... http://whereclause.com/blog/?p=83 ... Consuming XML in a stored procedure... http://whereclause.com/blog/?p=8 ... ... more

15

Santosh - 07/03/08

I want to insert records from XML string. Means my sql parameter will be XML data and in store procedure i ... more

Log in


Sign up | Forgot your password?

  • Chris Duckett IE9's H.264 vote killed Ogg

    In a split decision by the judges, the winner of the W3C/WHATWG video codec consensus is H.264, taking home the future of video playback on the internet while loser Ogg goes home with nothing but thoughts of what might have been. Read more »

    -- posted by Chris Duckett

  • Staff Google launches Apps Marketplace

    Google launches and app store, while Mozilla plans to re-write its open-source license. More of this week's news in the Roundup. Read more »

    -- posted by Staff

  • Staff Microsoft showcases new NUIs

    TechFest, Microsoft's internal even took place this week with researchers showcasing some new interfaces the company is working on. Read more »

    -- posted by Staff

Most popular tags

What's on?

  • Optus Deal

    Broadband + home phone + PlayStation®3 in a single package price!