-
Notifications
You must be signed in to change notification settings - Fork 51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
add support for source schema and exclude/include pattern and fix numeric type cast #28
Changes from 5 commits
1c41d74
3cfd4a8
d23ceb7
9fbcd21
d1708c6
8a446f3
aaebea4
9d43287
418ae11
fbe03e0
12d7032
6105325
3b05bf6
670c199
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,13 +24,20 @@ Or install it yourself as: | |
|
||
Set your source and target databases, as well as your s3 intermediary. | ||
|
||
*Note: Only set the POSTGRES_TO_REDSHIFT_EXCLUDE_TABLE_PATTERN if you want to exlude certain table(comma seperate for multiple value)* | ||
*Note: set DROP_TABLE_BEFORE_CREATE to true will drop the table on target before creation* | ||
|
||
```bash | ||
export POSTGRES_TO_REDSHIFT_SOURCE_URI='postgres://username:password@host:port/database-name' | ||
export POSTGRES_TO_REDSHIFT_TARGET_URI='postgres://username:password@host:port/database-name' | ||
export POSTGRES_TO_REDSHIFT_TARGET_SCHEMA='testing-data' | ||
export POSTGRES_TO_REDSHIFT_SOURCE_SCHEMA='testing-data' | ||
export S3_DATABASE_EXPORT_ID='yourid' | ||
export S3_DATABASE_EXPORT_KEY='yourkey' | ||
export S3_DATABASE_EXPORT_BUCKET='some-bucket-to-use' | ||
export DROP_TABLE_BEFORE_CREATE = true | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Also worth mentioning that this is false by default |
||
export POSTGRES_TO_REDSHIFT_EXCLUDE_TABLE_PATTERN = 'table-pattern-to-exclude1,table-pattern-to-exclude2' | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe comment here that all tables are included by default? |
||
export POSTGRES_TO_REDSHIFT_INCLUDE_TABLE_PATTERN = 'table-pattern-to-include1,table-pattern-to-include2' | ||
|
||
postgres_to_redshift | ||
``` | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -26,3 +26,33 @@ | |
end | ||
end | ||
end | ||
|
||
RSpec.describe 'a small source database with table to be excluded', type: :feature do | ||
context 'with a simple column' do | ||
before(:all) do | ||
PostgresToRedshift::Test.test_connection.exec(<<-EOS | ||
DROP TABLE IF EXISTS "p2r_integration"; | ||
DROP TABLE IF EXISTS "exclude_table"; | ||
CREATE TABLE IF NOT EXISTS "p2r_integration" ("id" SERIAL PRIMARY KEY, "title" text); | ||
INSERT INTO "p2r_integration" ("title") VALUES ('Casablanca'); | ||
CREATE TABLE IF NOT EXISTS "exclude_table" ("id" SERIAL PRIMARY KEY, "value" text); | ||
INSERT INTO "p2r_integration" ("value") VALUES ('Test'); | ||
EOS | ||
) | ||
end | ||
after(:all) do | ||
PostgresToRedshift::Test.test_connection.exec(%q[DROP TABLE IF EXISTS "p2r_integration";]) | ||
PostgresToRedshift::Test.test_target_connection.exec(%q[DROP TABLE IF EXISTS "p2r_integration";]) | ||
|
||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. nit: remove empty line here |
||
end | ||
|
||
it 'Copies all rows to target table' do | ||
PostgresToRedshift.update_tables | ||
result = PostgresToRedshift::Test.test_target_connection.exec( | ||
'SELECT * FROM "p2r_integration";' | ||
) | ||
expect(result.num_tuples).to eq(1) | ||
expect(result[0]).to eq('title' => 'Casablanca', 'id' => '1') | ||
end | ||
end | ||
end |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Worth mentioning that this is 'public' by default