Is it possible to use this program to extract cropped scans of only the cards I query? For example, I want to build a collage of all legal commander generals. Is it possible to query "Format Commander, Type Creature, and Type Legendary?"
Windows does not allow a directory named "CON" and if you follow the wikipedia list of MTG sets for set codes, the program will crash when it tries to save images for Conflux. Although this isn't entirely the fault of the program, it would be nice if it could throw a warning, or rename the directory (in some smart way) instead of crashing at that point. You could have it check the set names when a user hits save after editing them, before they start downloading data.
I dumped the full data of all sets last night (with "connectivity" to both sites) and I noticed three things:
Control/splitter characters used for the CSV need to be escaped in card text, or something. As of 01-27-2013, the flavor text of Magical Hacker is the only text that uses any of the default characters.
The rulings of Ertai, the Corrupted and Skyship Weatherlight are not captured. They were, three months ago. Probably Tahngarth, Talruum Hero's wouldn't extract either, if it had any. The foil version of these three cards had alternate art, that's the thing in common.
Gatherer Extractor can't fully open the resulting CSV file (13MB). It loads what seems like two thirds of it, then shows the error message "Database not formatted!".
I attacheduploaded two full dumps, the one from last night I mentioned, and another I did three months ago. Both can't be fully opened like I said. I'm on Windows 7 SP1 x64.
By the way, is there some prefered procedure to maintain a CSV with all sets, only by updating it, instead of redownloading everything each time? I redumped everything only because of the "Database not formatted!" error, I assume it can be done.
Windows does not allow a directory named "con" and if you follow the wikipedia list of MTG sets as the set code/directory names, the program will crash when it tries to save images for Conflux.
Nobody uses "CON" for exactly this reason. This is exclusively Microsoft's fault. The setcode for Conflux used everywhere I have seen is "CFX".
Nobody uses "CON" for exactly this reason. This is exclusively Microsoft's fault. The setcode for Conflux used everywhere I have seen is "CFX".
I understand that it's Microsoft's fault, and I realized why it was happening pretty quickly, but it's not true that no one uses it. If you want to pull pricing from say blacklotusproject.com they do use CON as can be clearly seen here: http://blacklotusproject.com/json/?cards=Ancient%20Ziggurat+(CON)
Since most webservers are running on Linux it's also not a problem to have a folder named 'CON' on your server, so it only becomes an issue when you try to create that folder on a Windows machine. Someone who isn't aware of this wouldn't know why the program is crashing. It's a pretty easy to fix from a user standpoint after grabbing the data, I just think the program should throw an error, since that would be user friendly.
Is it possible to use this program to extract cropped scans of only the cards I query? For example, I want to build a collage of all legal commander generals. Is it possible to query "Format Commander, Type Creature, and Type Legendary?"
Thanks!
I'm sorry but it's not actually. GEx is a tool to extract the whole MtG database, or partially based on sets only.
example : Renaissance, which magiccard.info code is "re", existing in FR and DE but not in EN.
2/ Do you plan to release your app in a open-source like, at least in way that other people can help you and work on this kind of feature ?
thx!
1/ Yes, but I have to look into more details about it. I will all let you know.
2/ Yes, but I have to investigate the easiest and safest way to do this. I don't have all the timeI want to develop GEx. I'm coding GEx and doing maintenance only after my day work or during the weekend on my spare time..., I really hope the MTG programmer community could maintain the project alive... even if the code has "decayed" since v3.0... I added so many things quite quickly, to fix some gatherer bugs etc. so that GEx recoded in a cleaner and more robust v4 version.
Gatherer Extractor can't fully open the resulting CSV file (13MB). It loads what seems like two thirds of it, then shows the error message "Database not formatted!".
I attacheduploaded two full dumps, the one from last night I mentioned, and another I did three months ago. Both can't be fully opened like I said. I'm on Windows 7 SP1 x64.
By the way, is there some prefered procedure to maintain a CSV with all sets, only by updating it, instead of redownloading everything each time? I redumped everything only because of the "Database not formatted!" error, I assume it can be done.
Nobody uses "CON" for exactly this reason. This is exclusively Microsoft's fault. The setcode for Conflux used everywhere I have seen is "CFX".
The alert "Database not formatted" is thrown when an error of data format is found in the csv. May you please compress the two files in another format? I can't open .7z files. A .zip or .rar file would be perfect I will check the files in debug mode to see what raises an error.
For minor sets, I usually reopen my .csv and add the new set. But when many wording or ruling have changed ( e.g. Comes into play ---> Enter the battlefield), then I usually extract all the data from scratch.
For promotional sets, lets say Friday Night Magic, I just load the .csv and overwrite the FNM set (don't forget to enable 'overwrite data' in options). All the previous cards will be updated and the new FNM cards will be added as well.
I will implement a warning if "CON" (protected in Win) is detected as a Set Code.
The alert "Database not formatted" is thrown when an error of data format is found in the csv. May you please compress the two files in another format? I can't open .7z files. A .zip or .rar file would be perfect I will check the files in debug mode to see what raises an error.
I assumed it was a red herring, but that's just because both files pass the python CSV parsing routines in my client script... I don't really know.
Sure, here you go. You should really migrate to 7Zip though, way better compression :tongue:.
Your two files are not correctly formatted because you use | as separator for your csv. Flavor of Niv-Mizzet from GP has | in it. Accordingly, the card exhibits an additional column, which raises the error.
You should definitely not change the default csv separator, set as || for that reason.
But the flavor text of Niv-Mizzet from Guildpact most certainly doesn't contain '|' or any other splitter character that I have noticed? Directly from both files:
#_"(Z–>)90º – (E–N²W)90ºt = 1"_#
I tried removing '²' and 'º', which no other card has, but the error persists.
Does anybody have old databases from Gatherer? Ones which would reflect the original wording from WotC and not the Oracle updated text? Even under the "Display: Printed" text tab some text is not accurate. It would be helpful to have a few versions, maybe one per year since Gatherer first launched.
Just a quick note, it looks like set code for magiccards.info is incorrect for Commander's Arsenal. It should be CMA, but it is CRS. Wouldn't it be nice to just have consistent set codes?
Just a quick note, it looks like set code for magiccards.info is incorrect for Commander's Arsenal. It should be CMA, but it is CRS. Wouldn't it be nice to just have consistent set codes?
Sure! Until the next release, you can modify it by yourself in the Set-code Editor. Just double click in the corresponding cell, and modify it. It should work afterwards.
But the flavor text of Niv-Mizzet from Guildpact most certainly doesn't contain '|' or any other splitter character that I have noticed? Directly from both files:
#_"(Z–>)90º – (E–N²W)90ºt = 1"_#
I tried removing '²' and 'º', which no other card has, but the error persists.
Yeah, you're right. I'm very sorry, I misred the debug results. Your error is coming from the flavor text of [UNH] Magical Hacker, which reads:
1|= y()u (4| r3@d 75, y0|_| /@ IVI0//$+3|2 &33|<
Because you're using '|' as separator for your csv, the corresponding row in your database has extra columns, raising the error. That is why by default, the separator is set as '||', and it is highly recommended not to modify it.
I just want to let you know that I temporary suspend the download of the Gatherer Extractor installer. This decision is correlated with the current legal disuptes concerning Cockatrice. Accordingly, I want to verify that GEx totally fits the last "Terms of Use" of the gatherer of Wizards of the Coast. I will remove all the subroutines that do not fit the legal environment imposed by WotC, and release the program again.
Everyone certainly easily understands that I don't want to be sued for a program I even don't make money from.
My main problem is that the WotC TOU is not really clear.
In the Terms of Use of the Gatherer Services, one can read, chapter 7:
7. User Content
As part of your use of the Services, you can create, include, refer to, transfer, upload, stream, display, host, promote, disseminate, post, make available, distribute, link to or transmit messages, artwork, text, displays, images, photographs, names, characters, items, graphics, screenshots, data, databases, information, files, pictures, video, audio, music, software and other files, items, materials and content, whether in written, digital, oral, machine-readable, electronic or visual form, to Wizards or our third-party servers of websites, or to other Users and/or their computers or networks, in various forms, such as in the selections you make for a service, posts, chat, and in forums and message boards, via a friends network and similar user-to-user areas, features and services which are not owned by Wizards, or to which Wizards does own rights or manages (collectively, your "User Content").
>> If you read carefully what is highlighted in blue, GEx seems perfectly legal, as it creates and disseminates to other users a database and pictures to which WotC does own rights and manages. But chapter 9 states:
9. Limits on Use of the Sites and Services
You agree not to engage in any of the following: (a) use any automated means, including, without limitation, agents, robots, scripts, or spiders, to access, monitor, data scrape, copy or transfer any part of the Sites or Services.
>> GEx is accessing the Services automatically, and copying part of the site. I think it violates this part of TOU.
(d) take any action that imposes an unreasonable or disproportionately large load on the infrastructure of the Sites or Services
>> What does "unreasonable" means ? If 1000+ ppl are using GEx at the same time, is the load on WotC servers unreasonable?
In addition:
8. Intellectual Property Ownership; Rights Granted to You
Site Content and the Services are the proprietary property of Wizards or its licensors or other Users, and are protected by copyright, trademark, and other intellectual property laws. You are granted a limited, non-sublicenseable license to access and use the Sites and Services, and to electronically copy (except where specified as prohibited) and print to hard copy portions of Site Content and/or Services for your personal, noncommercial use only; provided, that you preserve any copyright, trademark or other similar notices contained in or associated with such Site Content or Services. Such license is subject to these TOU and does not include: (a) any resale or commercial use of the Sites or any Site Content therein and/or Services; (b) the collection and use of any product or service listings, pictures or descriptions; (c) the distribution (electronic or otherwise), public performance, or public display of any Site Content or Services; (d) modifying or otherwise making any derivative uses of the Sites and Site Content or Services, or any portion thereof; (e) use of any data mining, robots or similar data gathering or extraction methods; (f) downloading (other than the page caching) of any portion of the Sites, any Site Content, the Services or any information contained therein, except as expressly permitted on the Sites or pursuant to the Services; or (g) any use of the Sites or Site Content or Services other than for its intended purpose. Any use of the Sites or Site Content or Services other than as specifically authorized herein, without the prior written permission of Wizards, is strictly prohibited and will terminate the license granted herein. This license is revocable at any time.
>> The first part tends to state that GEx is legal: it offers a way to electronically copy portions of the Site. In addition, all copyrights are preserved as GEx disclaimer clearly indicates that all copyrights, trademark are own by WotC. But the second part also clearly states that extraction methods are not included in the license granted.
I think one solution is to ask Wotc for a written permission to use the services as GEx does. And I am not really sure I'll be granted such a permission. The key point is that GEx is a really useful tool for handling a MtG database for boutique or fan websites, and is free of charge. The other solution is to extract database from magiccards.info only, despite I ignore if this is acceptable for WotC.
Sorry, I'm not good with those T&C...
I think as long as the program did not affect their sale of product (include Magic Online) should be alright just like magiccards.info
Well, I love the alternative idea of extracting form magiccards.info too.
Keep up the good work, I'm looking forward for your update.
Well I say that you should "manually" copy-and-paste the card data from gatherer page-by-page over the next week (wink). Save a "legal" copy of the database and include that on your own server. Then your application can merely download all the data from your server instead of the WotC server. This defeats the automated and unreasonable load limitations. The precompiled data could be downloaded by GEX users in record time. Most of your application code could be left unchanged, but still allow users to selectively save needed fields and file formats.
For section 9, I believe the intent is more geared towards site rips - ones that pull actual HTML, CSS and PHP from the site.
Section 8 is fine for you, since you are non-commercial. The data mining part is not applicable to your application (different kind of data mining intent there)
Section 7 basically grants you access to all the data to do most reasonable things with it, including making a copy of it. The "unreasonable" part is vague, intentionally, but not something you'd have to worry about. Even 1000 people using GEX wouldn't put a significant load on WotC servers.
well dang... i've been looking for a tool like this for such a long time, and once i finnaly find it, it has been discontinued.
I HOPE! with all my heart, that it is able to get back on its feet.
My collection is getting too large to keep track of, so i'm browsing the internet for the best ways to take inventory. This seems to be my ace card.
I use this software to keep my site updated, and I called Wizards about this and they said they shut down cockatrice because of the play testing aspect and the online play, that I could use this software without worrying about it.
Thanks all for your help.
anyway, "Fusion cards" of Dragon's Maze will probably need a local patch. But I have to know how it will be formatted on the gatherer.
This software is fantastic and I really hope you continue to update it.
I first noticed that this was not working when trying to update prices on a extractor database I had saved. I noticed that magiccards.info was not displaying pricing data and assumed it was connected with the current legal issues mentioned above.
However, I noticed today that the prices are back up and the software is able to update them again. I am hoping this is a good sign that magiccards.info does know or care about this softwares price updating feature.
Also from the recent post describing an update, does that mean this project is going to continue? I rely heavily on this software to keep my online store updated and may have to sell it if I can not continue to update it with ease.
Either way thanks for the great software. It really helped solve a lot of problems in the past!
Private Mod Note
():
Rollback Post to RevisionRollBack
To post a comment, please login or register a new account.
Is it possible to use this program to extract cropped scans of only the cards I query? For example, I want to build a collage of all legal commander generals. Is it possible to query "Format Commander, Type Creature, and Type Legendary?"
Thanks!
________________STANDARD________________
UBW Soorani Esper Control UBW
UBW Nicholas Spagnolo's Drownyard Control UBW
___________________EDH___________________
UB Wydwen, the Biting Gale UB
W Michiko Konda, Truth Seeker W
attacheduploaded two full dumps, the one from last night I mentioned, and another I did three months ago. Both can't be fully opened like I said. I'm on Windows 7 SP1 x64.By the way, is there some prefered procedure to maintain a CSV with all sets, only by updating it, instead of redownloading everything each time? I redumped everything only because of the "Database not formatted!" error, I assume it can be done.
Nobody uses "CON" for exactly this reason. This is exclusively Microsoft's fault. The setcode for Conflux used everywhere I have seen is "CFX".
I understand that it's Microsoft's fault, and I realized why it was happening pretty quickly, but it's not true that no one uses it. If you want to pull pricing from say blacklotusproject.com they do use CON as can be clearly seen here: http://blacklotusproject.com/json/?cards=Ancient%20Ziggurat+(CON)
Since most webservers are running on Linux it's also not a problem to have a folder named 'CON' on your server, so it only becomes an issue when you try to create that folder on a Windows machine. Someone who isn't aware of this wouldn't know why the program is crashing. It's a pretty easy to fix from a user standpoint after grabbing the data, I just think the program should throw an error, since that would be user friendly.
I'm sorry but it's not actually. GEx is a tool to extract the whole MtG database, or partially based on sets only.
1/ Yes, but I have to look into more details about it. I will all let you know.
2/ Yes, but I have to investigate the easiest and safest way to do this. I don't have all the timeI want to develop GEx. I'm coding GEx and doing maintenance only after my day work or during the weekend on my spare time..., I really hope the MTG programmer community could maintain the project alive... even if the code has "decayed" since v3.0... I added so many things quite quickly, to fix some gatherer bugs etc. so that GEx recoded in a cleaner and more robust v4 version.
The alert "Database not formatted" is thrown when an error of data format is found in the csv. May you please compress the two files in another format? I can't open .7z files. A .zip or .rar file would be perfect I will check the files in debug mode to see what raises an error.
For minor sets, I usually reopen my .csv and add the new set. But when many wording or ruling have changed ( e.g. Comes into play ---> Enter the battlefield), then I usually extract all the data from scratch.
For promotional sets, lets say Friday Night Magic, I just load the .csv and overwrite the FNM set (don't forget to enable 'overwrite data' in options). All the previous cards will be updated and the new FNM cards will be added as well.
I will implement a warning if "CON" (protected in Win) is detected as a Set Code.
Sure, here you go. You should really migrate to 7Zip though, way better compression :tongue:.
You should definitely not change the default csv separator, set as || for that reason.
If you have further issues, feel free to ask.
I tried removing '²' and 'º', which no other card has, but the error persists.
Sure! Until the next release, you can modify it by yourself in the Set-code Editor. Just double click in the corresponding cell, and modify it. It should work afterwards.
Yeah, you're right. I'm very sorry, I misred the debug results. Your error is coming from the flavor text of [UNH] Magical Hacker, which reads:
1|= y()u (4| r3@d 75, y0|_| /@ IVI0//$+3|2 &33|<
Because you're using '|' as separator for your csv, the corresponding row in your database has extra columns, raising the error. That is why by default, the separator is set as '||', and it is highly recommended not to modify it.
> Extraction of Watermark
> Correction of minor bugs
Enjoy
I just want to let you know that I temporary suspend the download of the Gatherer Extractor installer. This decision is correlated with the current legal disuptes concerning Cockatrice. Accordingly, I want to verify that GEx totally fits the last "Terms of Use" of the gatherer of Wizards of the Coast. I will remove all the subroutines that do not fit the legal environment imposed by WotC, and release the program again.
Everyone certainly easily understands that I don't want to be sued for a program I even don't make money from.
Where do i get the Gatherer Extractor executable.
Http://www.mediafire.com/?3akc6mtapnar2 The link is not working.
The 3.6 version is not downloading the information of power and toughness.
thank you
In addition, 3.6 is obsolete.
I miss it...
In the Terms of Use of the Gatherer Services, one can read, chapter 7: >> If you read carefully what is highlighted in blue, GEx seems perfectly legal, as it creates and disseminates to other users a database and pictures to which WotC does own rights and manages. But chapter 9 states: >> GEx is accessing the Services automatically, and copying part of the site. I think it violates this part of TOU. >> What does "unreasonable" means ? If 1000+ ppl are using GEx at the same time, is the load on WotC servers unreasonable?
In addition: >> The first part tends to state that GEx is legal: it offers a way to electronically copy portions of the Site. In addition, all copyrights are preserved as GEx disclaimer clearly indicates that all copyrights, trademark are own by WotC. But the second part also clearly states that extraction methods are not included in the license granted.
I think one solution is to ask Wotc for a written permission to use the services as GEx does. And I am not really sure I'll be granted such a permission. The key point is that GEx is a really useful tool for handling a MtG database for boutique or fan websites, and is free of charge. The other solution is to extract database from magiccards.info only, despite I ignore if this is acceptable for WotC.
Any help to clarify those issues is very welcome.
I think as long as the program did not affect their sale of product (include Magic Online) should be alright just like magiccards.info
Well, I love the alternative idea of extracting form magiccards.info too.
Keep up the good work, I'm looking forward for your update.
Section 8 is fine for you, since you are non-commercial. The data mining part is not applicable to your application (different kind of data mining intent there)
Section 7 basically grants you access to all the data to do most reasonable things with it, including making a copy of it. The "unreasonable" part is vague, intentionally, but not something you'd have to worry about. Even 1000 people using GEX wouldn't put a significant load on WotC servers.
I HOPE! with all my heart, that it is able to get back on its feet.
My collection is getting too large to keep track of, so i'm browsing the internet for the best ways to take inventory. This seems to be my ace card.
GL Chaudakh!
anyway, "Fusion cards" of Dragon's Maze will probably need a local patch. But I have to know how it will be formatted on the gatherer.
I will let you know
I first noticed that this was not working when trying to update prices on a extractor database I had saved. I noticed that magiccards.info was not displaying pricing data and assumed it was connected with the current legal issues mentioned above.
However, I noticed today that the prices are back up and the software is able to update them again. I am hoping this is a good sign that magiccards.info does know or care about this softwares price updating feature.
Also from the recent post describing an update, does that mean this project is going to continue? I rely heavily on this software to keep my online store updated and may have to sell it if I can not continue to update it with ease.
Either way thanks for the great software. It really helped solve a lot of problems in the past!