Skip to main content

How to load huge amount of data from csv to postgre

Recently I got a defiance from a some developers regarding postgresql server is slow performance. In his opinion it is not able to fetch 5 lac record from CSV files. They show there aplication which was using (node.js+sequlize ORM). When executed it was really sucking in middle.

5 lac records are really in huge amount but postgre is alos being for high performance application. There is no question we can say that it can not able to fetch this amount of data. So I suggested him to start importing from short amount of records like top 6 then top 600 then top 6000 then 60000. Yeah he did and really it stopped after 60000 records. It failed from application, program is running, running and keep running..

So boll goes in my bucket then I started digging on database server configuration parameters like effective_cache_size, work_memory, Shared_buffers, Maximum_number_Connections, wal_buffers etc, given appropriate values as per best practices and current system resources. Then restart the server to try again.

We repeated the previous actions like insert in short amount like top 6 then top 600 then top 6000 then 60000. Yeah he did and really it stopped after 60000 records. Again it was fail from application.

Now I tried to insert from database console. I executed the below query and you will not believe it finished in less then 2 seconds with all data inserted into table.

CREATE TABLE fare
(
CompanyCode character varying(255),
LineNumber character varying(255),
CardTypeCode character varying(255),
FirstLocationCode character varying(255),
SecondLocationCode character varying(255),
FareAmount character varying(255)
)

copy fare(CompanyCode,LineNumber,CardTypeCode,FirstLocationCode,SecondLocationCode,FareAmount)
From E'C:\\master_part\\cmn0006_0103.tsv' with (format csv, delimiter E'\t')

delete from fare where CompanyCode='Company Code'

insert into "fareMaster"("companyCode","lineNumber","cardTypeCode","firstLocationCode","secondLocationCode","fareAmount")
select companycode,linenumber,cardtypecode,firstlocationcode,secondlocationcode,fareamount from fare ;




So ball is out of my court and when he started digging to application there are some minor changes he did at ORM side and application started fetching lacs in database. Now he believes how postgre can handle huge data insert simultaneously.

Comments

Popular posts from this blog

mongoDB error : aborting after fassert() failure

What to do when facing errors on mongoDB “aborting after fassert() failure”

I like errors, in mongoDB this is the first error I faced and luckily many times. This error i faced during restoring name-space on local and restarting db system. I am still searching the exact root cause of this issue but i am able to resolve the current problem through below steps.

Remove all relevant namespace files from data-file route path..Now repair mongo instance using mongod process.mongod --repair ////////// execute command from bin folder path Then start server using mongd process, if started server successfully then ..mongod  ////////// execute command from bin folder path Restore last backups as normal process.Now check database by connecting mongo shell. Thanks for reading, 
Please comment your experience if you faced and also share knowledge if you have better steps to resolve...


https://www.facebook.com/pages/Sql-DBAcoin/523110684456757

SQL71562: external references are not supported when creating a package from this platform

Last week I got this error from one of developer who was trying to deploy his project from Testing server to SQL Azure QA server. He was using “Deploy Database to SQL Azure” option from SSMS Tool-Task option.

After connecting to SQL Azure portal when operation started to deployment below errors occurs.

Validation of the schema model for data package failed. Error SQL71562: Error validating element xx.xxx.xx:function .dbo.xxx has an unresolved refrence to object xx.dbo.xxxx external refrences are not supported when creating a package from this platform.



Reason: The reason of the this error was; some functions of project was dependent on master database and only single database was being deploy to SQL Azure. DACFx must block Export when object definitions (views, procedures, etc.) contain external references, as Azure SQL Database does not allow cross-database external references So, this error was coming.

Solution : I suggested him to create those function to locally on local database what…

How to add an article in Transactional Replication

If we have a set-up of Transactional Replication for Data Distribution running and wanting to add new object to replication on other server we can follow below process.
To add an article In Transaction replication with PUSH Subscription