01:10
heckacopter_ joined
01:39
<suprball>
umm is it OK to ask a quick question here? ;<
01:39
<alxgsv>
you just did!
01:41
<suprball>
Well, I'm just wondering (and I've googled, but hasn't found an answer) if the php-driver for mongo supports the "_addSpecial"-stuff?
01:43
<suprball>
(since I need to specify maxScan and similar stuff)
01:46
bulatshakirzyano joined
02:25
<skot>
yeah, you want this : http://us.php.net/manual/en/mongocursor.addoption.php
03:18
sjaak_trekhaak joined
03:55
GabrielVieira joined
04:06
niftylettuce_ joined
05:03
abstrusenick joined
05:05
<remonvv>
Hi, I can't quite figure this out based on the code; does a MongoDB write lock prevent reads? Does it disallow the acquisition of read locks when it has a write lock?
05:06
<MacYET>
it does not
05:47
<remonvv>
Right, that's what I thought. But that makes this "The mongod process uses a modified reader/writer lock with dynamic yielding on page faults and long operations. Any number of concurrent read operations are allowed, but a write operation can block all other operations." a rather confusing statement.
05:49
<remonvv>
Well, actually, what I thought is that it followes standard RW lock semantics (1 concurrenr write, read lock acquisition is blocked until write lock is released, and reads can happen concurrently).
05:53
<Lujeni>
remonvv: What's is your question remonvv ?
05:53
<remonvv>
If MongoDB write lock prevents reads during it.
05:54
<remonvv>
The documentation implies it does.
05:54
<Lujeni>
remonvv: To my mind, use replica set with query/read on slave it's best way
05:55
<remonvv>
Well, that has nothing to do with the question though. The documentation implies a write lock blocks "all operations". I do not think that is true. Rather it blocks operations that also require a write lock.
05:56
<remonvv>
I was just curious if I misunderstood.
05:56
<MacYET>
it must be hard reading docs the wrong way
05:57
<ron>
not as hard as it is to write the docs the wrong way.
05:57
<remonvv>
You can't read documentation the wrong way, you can only write them the wrong way.
05:57
<remonvv>
Oh, great minds.
05:57
<remonvv>
Not strictly true ofcourse, but generally true.
05:58
<remonvv>
Confusion is usually the result of vague or ambiguous writing rather than questionable reading.
06:48
<Cancerus>
what is "machine id"?
06:48
<MacYET>
documented in the ObjectId docs
08:03
Error404NotFound joined
08:14
nathenharvey_ joined
08:17
<imran_>
can you do nested queries in mongodb without specifying the intermediate keys ?
08:17
<imran_>
for for example if you had the doc {
08:17
<imran_>
'a': {' score': 10 }
08:17
<imran_>
'b': {'score':20}
08:17
<MacYET>
no you can't
08:18
<imran_>
could you do a query that returned all documents which had fields which had fields which matched a criteria (score > x)
08:18
<MacYET>
didn't I say no?
08:18
<imran_>
ok, just making sure it was clear what I was saying :)
08:19
<* MacYET>
shakes head
08:32
<balboah>
is the count on a collection supposed to jump up/down while mongos is rebalancing shards?
08:37
<skot>
it can, depending on a few factors. How are you counting?
08:38
<skot>
Are you using a query?
08:39
<skot>
Here is an issue you might want to track: https://jira.mongodb.org/browse/SERVER-3645
08:46
<remonvv>
skot, any idea what this is "combinatorial limit of $in partitioning of result set exceeded"? Triggered at line 917 in queryutil.cpp
08:46
<remonvv>
We're hitting it with relatively small $in clauses (1000-1300 elements)
08:47
<remonvv>
Not familiar enough with the code to know what "size" refers to in uassert( 13385, "combinatorial limit of $in partitioning of result set exceeded", size() < 1000000 );
08:53
<balboah>
skoef: I'm simply doing db.collection.count() in the mongos
08:59
<skot>
balboah: yep, that is the bug/improvement
08:59
<balboah>
skot: good to know, thanks
08:59
<skot>
remonvv: no, and i'm running out the door, can you post the query form?
08:59
<skot>
Are you using more than one $in in the query?
09:00
<skot>
(and what version?)
09:02
<remonvv>
2.1, one $in, objectId list
09:02
<remonvv>
it's literally {a:1, b:{$in:[ObjectId(...), ...]}}
09:02
<remonvv>
same issue on 2.01
09:03
<remonvv>
It's to a sharded collection, if that changes anything
09:04
<remonvv>
On non-sharded it doesn't seem to happen. But I can't really think of a reason why that would make a difference so I think that might be coincedence.
09:11
<remonvv>
Oh, it is sharding related by the looks of it.
09:11
<remonvv>
https://jira.mongodb.org/browse/SERVER-4778
09:18
napperjabber joined
09:23
corruptmemory joined
09:52
SkyRocknRoll joined
09:52
SkyRocknRoll joined
10:03
JaredMiller_ joined
10:28
<Tankado>
Hello all, I am new to DB, i am reading alot about mongo and i sometimes see about saving entrie documents, i also read about BSON and saw the syntax but i dont understand how a text document can be index in MongoDB ?
10:28
<MacYET>
mongodb has no fulltext search
10:29
<MacYET>
next in line
10:32
<Tankado>
Ok and is there any serialization\deserialization client of C++ objects, similar to
10:32
<Tankado>
GSON (Google java serialization to JSON)
11:07
<asciiduck>
I'm wirting something to parse mongodb's server status and put it into Graphite, does MongoDB support any type of index besides btree? (just curious as the indexCounters object has a subobject of 'btree')
11:09
<skot>
no, just btree.
11:14
<jwage>
What would a better more efficient way to do this query? {"metadata.objectHistory":{"$exists":true}} ?
11:15
<jwage>
maybe this? {"metadata.objectHistory":{"$ne":null}}
11:16
thomasdebenning joined
11:18
<jwage>
skot, any suggestions on the above query?
11:23
roflcerofthelawl joined
11:26
napperjabber joined
11:28
<remonvv>
If there's an index on that metadata or metadata.objectHistory then the latter is more efficient. If not it doesn't matter.
11:28
<remonvv>
.explain() could've told you that ;)
11:29
<balboah>
I'm getting this when trying to shard a collection: exception: invalid parameter: expected an object ()
11:29
<balboah>
I have no clue of what that means as the same command works on another database with the same structure
11:29
<remonvv>
Paste your entire command, looks like a syntax error
11:30
<balboah>
remonvv: http://pastebin.com/6rHumnSM
11:34
<balboah>
remonvv: any idea of what may be the cause when same command (and same index keys but different values) works for a different database?
11:34
<remonvv>
Looks okay, db.version()?
11:35
<balboah>
it's 2 shards running 2.0.2 on linux debian
11:35
<balboah>
except for mongos which runs on freebsd
11:36
<remonvv>
Hm, not a clue. The command is valid.
11:36
<remonvv>
Tried "shardcollection"?
11:36
<remonvv>
Maybe there's a weird case sensitivity issue on that build.
11:36
<remonvv>
I just ran it on 2.0.1 and 2.1
11:36
<remonvv>
Both work
11:39
<balboah>
remonvv: no go, except for also lowercasing the collection but then it created an empty one
11:40
<balboah>
maybe I'll just report it
11:45
<remonvv>
Yeah there's nothing wrong with the command so probably a OS specific bug.
11:46
thewildpendulum joined
11:50
<balboah>
remonvv: reported. Had the same problem trying a linux mongos
11:50
andreassaebjoern joined
11:51
<remonvv>
Hm, well we run all our stuff on that and no issues so far.
11:51
<balboah>
yeah I have 5 databases where all but one works
11:54
<thewildpendulum>
i'm trying to store backbones models using mongoose and all the code i've read so far isn't very straightforward about how this actually done. is it as simple as opening a connection and calling save() or do i need to define a sync method?
11:58
<IAD>
if i need work with unknown count of values of collection(limit by script time execution), prefered use iterator or findOne a lot times? I have 100-500 iterations per one script execution.
11:58
<Tankado>
What kind of types can serve as "indexes" in MongoDB ? from last response i get it i cant use and search string indexes
11:59
<MacYET>
what kind of types?
12:01
heckacopter_ joined
12:02
<remonvv>
Tankado, you can use strings fields as (part of) an index.
12:02
<remonvv>
IAD, use a cursor + getmore()
12:03
<remonvv>
getmore is almost always wrapped in the driver by an iterator, so yes, iterator.
12:06
<Tankado>
yeah i get it now, sorry for the stupid question.
12:06
<remonvv>
no problem ;)
12:07
<wtpayne2>
Hi. I am trying to figure out how best to handle AutoReconnect exceptions.
12:07
<wtpayne2>
I assume I have to wait for a bit before trying the next query.
12:08
<wtpayne2>
Any idea what a suitable period might be?
12:10
<remonvv>
What language/platform are you talking about?
12:10
<remonvv>
You're either connected or you're not.
12:11
<remonvv>
And it doesn't give you any indication the connection has been reestablished?
12:11
<wtpayne2>
I have not encountered any problems yet,
12:11
<remonvv>
Are you sure this is actually an issue you have to work around?
12:11
<wtpayne2>
But the AutoReconnect exception must exist for some reason.
12:11
<remonvv>
It should maintain a large pool of connections and only use ones that are, you know, connected
12:12
<remonvv>
It exists so you can know that the operation you were trying to do might (and probably will have) failed
12:12
<wtpayne2>
So any exceptions that I get (other than StopIteration) are genuine errors.
12:13
<remonvv>
I'm not a pymongo expert, but every unique exception should be there for a reason and should be dealt with as is documented. But yes, generally speaking that's a fair assumption.
12:13
<remonvv>
It shouldn't raise exceptions if it can recover by itself.
12:14
<wtpayne2>
Well, as far as I am aware, the AutoReconnect exception exists to let you know (as you stated) that the previous operation might have failed.
12:14
<wtpayne2>
So you can then decide whether to retry, or to just press on regardless.
12:14
<remonvv>
Yes, so it can't recover by itself.
12:14
<remonvv>
Mind you, for w > 0 writes it should be able to recover but hey
12:15
<wtpayne2>
Ok. So do you need to reconnect if the AutoReconnect exception is thrown?
12:15
<remonvv>
Doubtful. It'll do the reconnecting. The exception is just there to make you aware that whatever you were doing might have failed.
12:15
<wtpayne2>
I.e. disconnect() the current Connection and try to create a new Connection()
12:15
<wtpayne2>
Ok. That was my understanding.
12:15
<IAD>
remonvv: http://
www.mongodb.org/display/DOCS/Queries+and+Cursors thay write "If the result set is large, finding the first 100 results may be much faster than finding all results and printing the first 100."
12:16
<remonvv>
Not sure. Not a pymongo user. Having to do it manually would be rather stupid though.
12:16
<wtpayne2>
Yeah, that's what I thought.
12:16
<wtpayne2>
I think that I can just press on with the next query.
12:16
<wtpayne2>
It seems prudent to insert a pause to let (whatever might have caused the previous query to fail) to correct itself.
12:16
<remonvv>
IAD, obviously. If you use skip(..) the server will iterate over all results but will not return them. Iterating over all documents on the client-side will force the server to send all documents to the client.
12:17
<remonvv>
So if you need the 10000 to the 10010th document you should blindly query it all.
12:17
<remonvv>
But that's not what you asked. You asked if it's more efficient to do single findOne's versus find() + iterator.
12:17
<remonvv>
The latter is more efficient.
12:17
<remonvv>
wtpayne2, the pause would be completely arbitrary
12:17
<IAD>
remonvv: i use many times: $this->collection->find($where)->sort($sort)->limit(1)->getNext();
12:18
<remonvv>
It should start using a different connection for all subsequent operations
12:18
<remonvv>
so use a cursor instead
12:18
<remonvv>
yeah doesn't matter then
12:19
<remonvv>
findOne() == find().limit(1)
12:19
<IAD>
findOne used without sort
12:20
<wtpayne2>
Ok. I guess some errors would be caused by network glitches, so a short pause at first would be OK, but then again, the error might be caused by the server going down, so a longer pause (poll every five minutes or so) after a while would be good.
12:24
<remonvv>
? you can't findOne without sort and find with sort and get the same result
12:25
<IAD>
yes, i can't findOne with sort. Not find it in man
12:27
<IAD>
remonvv: http://groups.google.com/group/mongodb-user/browse_thread/thread/4fb4e270a8da35f1 no way =)
12:31
nathenharvey_ joined
12:33
Error404NotFound joined
12:39
andreassaebjoern joined
13:08
scottbessler joined
13:11
scottbessler joined
13:13
scottbessler joined
13:26
nathenharvey_ joined
13:28
corruptmemory joined
13:28
nathenharvey_ joined
13:28
SkyRocknRoll joined
13:58
GabrielVieira joined
14:20
<phlff>
I stupidly just blew away my test DB before asking this question, but I have a dev env and even though there is usually only a few hundred k of data in it, my DB directory quickly grows to about 3.5 GB. Is this normal and is it configurable in the settings?
14:21
<geoffeg_c>
phlff: it's probably your database files being pre-allocated
14:22
<geoffeg_c>
phlff: http://
www.mongodb.org/display/DOCS/Excessive+Disk+Space#ExcessiveDiskSpace-DatafilePreallocation
14:23
<phlff>
geoffeg_c: Well that explains it, I believe. Thanks.
14:29
<phlff>
geoffeg_c: Just tested. rm'ing the data dir then restarting mongo makes the data dir 413MB. On first run of my application which creates the new collections but hasn't actually entered in any data, I see disk usage jump to 3.3GB. So you are right, it is simply preallocation of the data files.
14:39
<HardFu>
how can I concatenate numbers from a field_
14:39
<HardFu>
for example { count: 3 }
14:39
<HardFu>
and I'd like to get a total count for all records
14:39
<HardFu>
each one would add as much as the count value is
14:46
<Pedro>
is it recommended to shard a collection by its _id? most of the queries are using the _id as a key?
14:46
<vadi>
> print ( ISODate("2010-07-30T15:05:00Z") );
14:46
<vadi>
Fri Jul 30 2010 16:05:00 GMT+0100 (GMT)
14:47
<vadi>
How to get the actual "2010-07-30T15:05:00Z" value?
15:02
<asciiduck>
Is there a way to record all the queries a MongoDB server receives soI can replay them? DiagLogging seems to only record some of the reads.
15:20
<vadi>
How to covert ISODate("2010-07-30T15:05:00Z")
15:21
<vadi>
to ISO8601 format?
15:26
<skot>
asciiduck: No, not really. You could put a proxy in front to record them.
15:26
<skot>
You can turn on database profiling to collect all operations, and just filter out the queries.
15:26
<skot>
But there are limitations of the size of queries recorded there.
15:29
<vadi>
thanks mattbillenstein . Let me look
15:51
<asciiduck>
When I enable profiling on a database in a replica set, does taht store profiling data
15:51
<asciiduck>
Does that store profiling data for the whole replica set in the collection?
15:52
<skot>
no, just the local server instance.
15:57
<vadi>
Solved, thanks mattbillenstein
15:57
<asciiduck>
Okay, thanks.
16:10
<mattbillenstein>
vadi: np
16:25
sr71-blackbird joined
16:36
<moshef>
guys, its not possible to search within a nested documents with or, right?
16:36
<moshef>
i remember asking about it few weeks ago and i was directed to a feature in development page regarding this
17:02
napperjabber_ joined
17:08
napperjabber joined
17:29
<skot>
moshef: do you have an example? you can use $or with dot-notation no problem.
17:34
corruptmemory joined
17:43
<bsegal>
jwage: you around to take a look at a doctrine question on SO?
17:45
<bsegal>
http://stackoverflow.com/questions/9030901/doctrine-odm-set-random-value-generated-by-js-math-random-on-pre-persist
17:46
<jwage>
unfortunately i dont think this is possible right now
17:46
<bsegal>
this isn't my question, but i can relay that to them if you want
17:47
<jwage>
it depends on what type of field random is mapped as actually
17:48
dudewheresmycode joined
17:49
<dudewheresmycode>
is there a way to do a SUM of a field in a query.. e.g. db.col.find({id:1},{$sum:{count:1}}) or something?
17:49
<bsegal>
jwage: i see. thanks for having a look.
17:50
<skot>
dudewheresmycode: you can use group or map/reduce or the new aggregation framework (in the current dev nightly)
17:51
<dudewheresmycode>
but will that give me the current sum? or just the last nightly map reduce when i query it?
17:52
<dudewheresmycode>
can i simply run the map reduce right before (or along with) each query?
17:54
<skot>
A better thing is to keep a counter with the sum so it is a constant time operation.
17:54
<skot>
Then anytime you add/remove from a count, you can keep the sum up to date.
18:06
<dudewheresmycode>
some more specifics on the project.. their for analytics.. so there's a count (or sum) field per record kept by hour.. so 1am = 50, 2am = 59, 3am = 34
18:07
<dudewheresmycode>
i need to query and ask for the sum of a certain time period, say the last 24 hours.. so i query the time fields and it returns me 3 records with the {count} fields and i have to loop thru the results to sum up the total time period
18:07
<nick_h_>
with docs like this: {name: "Bob", things: [ {name: "team", value: "vikings"}, {name: "colour", value: "red"} ] } is there a way to sort the docs on the "value" field where name is "team"?
18:07
johnanderson joined
18:09
<dudewheresmycode>
db.collection.find( { "name": "team" } ).sort( { "value": 1 } )
18:09
<dudewheresmycode>
^^^^ — RE: nick_h_
18:10
<nick_h>
dudewheresmycode: that'll sort on the "name" field in the root doc. i'm trying to sort on the "name" field in the documents in the "things" field. ie: things.name
18:10
<nick_h>
oops, let me rephrase that
18:11
<nick_h>
i'm trying to sort on "things.value", but only where "things.name" is "team"
18:11
<dudewheresmycode>
you might try using the $in operator
18:13
<nick_h>
dudewheresmycode: i'm not sure how $in would help. that's for restricting which documents are matched, rather than for sorting
18:15
<dudewheresmycode>
and you can't just do .sort({things:{name: 1}}) ?
18:17
<dudewheresmycode>
db.collection.find( {"things": { "value": "vikings" } } ).sort( { "things": { "name": 1 } } )
18:17
<dudewheresmycode>
not tested.. but why wouldnt that work?
18:20
<nick_h>
dudewheresmycode: "things" is an array. it's not possible to sort on an array. i'm trying to sort by a field in one specific doc within the "things" array
18:21
<dudewheresmycode>
map reduce?
18:21
<nick_h>
that might work
18:21
<Ry4an>
the new aggregations stuff would do that, but it's only in development builds now
18:22
<dudewheresmycode>
or just do it client side for now.. unless you're dealing with tons of data
18:22
<dudewheresmycode>
return what you can and create a custom script for sorting in your app
18:22
<dudewheresmycode>
what language are you working in? or you doing this via mongo shell?
18:25
<nick_h>
i'm doing this in Ruby using Mongoid. however, there's too much data to retrieve it all and sort client-side
18:32
<tornd>
hm, anyone that can shed some light on this?
18:32
<tornd>
https://gist.github.com/1691573
18:32
<tornd>
querying on one field and sorting on two fields, all 3 fields indexed, yet it performs a scan of sorts
18:32
<tornd>
not a full scan, but still
18:33
<tornd>
reading the index docs here I should never see nscanned larger than 0, and indexOnly should be true, if a query only hit the index
18:36
<tornd>
or would it be better design to just index the instance_id field here and not index the fields I'm sorting on? the amount of users per instance_id won't grow especially large, the docs mentions it being doable on result sets less than 4MB
18:39
thomasdebenning joined
18:40
<tornd>
hm, maybe I'm just misreading the explain() docs here
18:52
<skot>
looks good, nscanned == n which is what you return.
18:53
<skot>
indexOnly refers to if the query doesn't need to return data from the documents, like the data is in the indexes.
18:55
<tornd>
skot: yeah, figured - just misread the explain docs
18:56
<tornd>
should I assume I'm in the clear as long as it chooses a BtreeCursor cursor and nscanned == n ?
18:58
<skot>
That is the best you can do.
19:02
<tornd>
allright, makes more sense then
19:02
<tornd>
did the massive trick of coding for 3 days straight not caring about indexes since I hadn't read up properly on it yet, thinking I'd go through the model code and add them later
19:02
<tornd>
note fo self and so on
20:54
GabrielVieira joined
20:56
roflcerofthelawl joined
21:46
avalanche123 joined
22:09
dudewheresmycode left
22:50
<veg^>
is there way to pass $ like escaped symbol or smth, that it will be treated as normal $....
22:56
<alnewkirk>
veg^: thats gonna be a BIG why?
22:56
<* alnewkirk>
suspects veg^ is asking for trouble
22:58
<veg^>
i need to check operators, that pass to engine, i check it for q[0] == '$'
22:59
<alnewkirk>
veg^: that statement is ambiguous
23:01
<veg^>
so any hints?
23:01
johnanderson joined
23:03
<alnewkirk>
veg^: can you show me an example of what you're trying to insert?
23:04
<veg^>
i'm checking for operators in white list, i saw here http://
www.mongodb.org/display/DOCS/Do+I+Have+to+Worry+About+SQL+Injection that i can use utf8 for $, right
23:05
<veg^>
now i just checking for symbol '$', is it enough
23:06
<veg^>
i'm using php if it makes sense
23:07
<alnewkirk>
simply put, just dont
23:20
avalanche123 joined
23:31
<Goopyo>
Anyone know read through a java result?
23:32
<Goopyo>
not sure how to move through/process the data returned
23:33
<veg^>
why "\uFF04" doesn't work for "$", works only "
23:39
<veg^>
but here is mentioned \uFF04 http://
www.mongodb.org/display/DOCS/Do+I+Have+to+Worry+About+SQL+Injection
23:39
<veg^>
i checked in the sheel
23:56
<veg^>
also why should i worry about '.' in keys, what is it doing?
23:59
<alnewkirk>
wtf veg^