[Davical-general] CardDAV read and write issues from Apple clients

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

[Davical-general] CardDAV read and write issues from Apple clients

David Newman
DAViCal 1.1.2 installed from git with Ján Máté's 2012-11-01 and
2013-04-09 patches.

Append works now for importing large address books. Thanks Ján!

This server now has an address book with nearly 6,000 entries, imported
in two parts using the append feature. The number of contacts on the
server agrees with the number exported from Contacts.app on OS X 10.8.3.
Also, I granted my server account all privileges for this resource.

However, there are issues sharing contacts across platforms:

1. READ: After adding the DAViCal server as an contacts account,
Contacts.app on OS X sees only 714 of the ~6,000 entries.

After trying to retrieve entries from the server, the client app
eventually displays an error icon (a triangle with an exclamation
point). Clicking this icon returns this message:

The operation couldn’t be completed. (CoreDAVErrorDomain error 3.)

Google is no help with that error code.

Similarly, an iPhone and iPad both running iOS 6.1.3 both see only 1,214
of the ~6,000 entries. There's no error displayed.

2. WRITE: Adding a contact on either iOS device is possible, but the
contact is not added to the DAViCAL server.

Also, the added contact disappears after navigating away from the
Contacts app. I don't just mean disappears from the server's group; I
mean the newly added contact doesn't appear under any group, even though
it's previously been saved.

Similar issue with OS X; I can add a new contact, save it, and drag it
onto the server list,  but the number of server entries does not increase.

Neither 1 nor 2 above produce an error on the server side*, and I've
verified there's nothing on the firewall between client and server that
is blocking communications.

Thanks in advance for troubleshooting clues.

dn

*There are some entries in the error logs that pertain to the address
book; samples below. However, these entries are from *before* attempting
the read and write problems described above.

[Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
REPORT: Query: SQ: Took: 0.344631 for SELECT collection.*,
calendar_item.*, caldav_data.*, addressbook_resource.*, sync_changes.*
FROM collection LEFT JOIN sync_changes USING(collection_id) LEFT JOIN
caldav_data USING (collection_id,dav_id) LEFT JOIN calendar
[Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
REPORT: Query: SQ: _item USING (collection_id,dav_id) LEFT JOIN
addressbook_resource USING (dav_id) WHERE collection.collection_id =
:collection_id AND sync_time >= (SELECT modification_time FROM
sync_tokens WHERE sync_token = :sync_token) ORDER BY collection
[Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
REPORT: Query: SQ: .collection_id, sync_changes.dav_name,
sync_changes.sync_time







------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Ján Máté-2
Your problem is very probably OS X/iOS client timeout. The biggest request from the client is the "multiget report" which returns all vCards stored on the server (~6000 entries = the response is maybe too large?) ...


JM

On Apr 14, 2013, at 7:32 PM, David Newman <[hidden email]> wrote:

> DAViCal 1.1.2 installed from git with Ján Máté's 2012-11-01 and
> 2013-04-09 patches.
>
> Append works now for importing large address books. Thanks Ján!
>
> This server now has an address book with nearly 6,000 entries, imported
> in two parts using the append feature. The number of contacts on the
> server agrees with the number exported from Contacts.app on OS X 10.8.3.
> Also, I granted my server account all privileges for this resource.
>
> However, there are issues sharing contacts across platforms:
>
> 1. READ: After adding the DAViCal server as an contacts account,
> Contacts.app on OS X sees only 714 of the ~6,000 entries.
>
> After trying to retrieve entries from the server, the client app
> eventually displays an error icon (a triangle with an exclamation
> point). Clicking this icon returns this message:
>
> The operation couldn’t be completed. (CoreDAVErrorDomain error 3.)
>
> Google is no help with that error code.
>
> Similarly, an iPhone and iPad both running iOS 6.1.3 both see only 1,214
> of the ~6,000 entries. There's no error displayed.
>
> 2. WRITE: Adding a contact on either iOS device is possible, but the
> contact is not added to the DAViCAL server.
>
> Also, the added contact disappears after navigating away from the
> Contacts app. I don't just mean disappears from the server's group; I
> mean the newly added contact doesn't appear under any group, even though
> it's previously been saved.
>
> Similar issue with OS X; I can add a new contact, save it, and drag it
> onto the server list,  but the number of server entries does not increase.
>
> Neither 1 nor 2 above produce an error on the server side*, and I've
> verified there's nothing on the firewall between client and server that
> is blocking communications.
>
> Thanks in advance for troubleshooting clues.
>
> dn
>
> *There are some entries in the error logs that pertain to the address
> book; samples below. However, these entries are from *before* attempting
> the read and write problems described above.
>
> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
> REPORT: Query: SQ: Took: 0.344631 for SELECT collection.*,
> calendar_item.*, caldav_data.*, addressbook_resource.*, sync_changes.*
> FROM collection LEFT JOIN sync_changes USING(collection_id) LEFT JOIN
> caldav_data USING (collection_id,dav_id) LEFT JOIN calendar
> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
> REPORT: Query: SQ: _item USING (collection_id,dav_id) LEFT JOIN
> addressbook_resource USING (dav_id) WHERE collection.collection_id =
> :collection_id AND sync_time >= (SELECT modification_time FROM
> sync_tokens WHERE sync_token = :sync_token) ORDER BY collection
> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
> REPORT: Query: SQ: .collection_id, sync_changes.dav_name,
> sync_changes.sync_time
>
>
>
>
>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

David Newman
On 4/14/13 10:59 AM, Ján Máté wrote:
> Your problem is very probably OS X/iOS client timeout.

Based on the spinning wheel icon on both client types, I'd tend to agree.

Is there anything I can do on the server or client side to reduce the
size of the request? (That is, other than the obvious solution of
breaking one big address book into multiple smaller ones...which still
would involve lots of responses from the server.)

Thanks!

dn


The biggest request from the client is the "multiget report" which
returns all vCards stored on the server (~6000 entries = the response is
maybe too large?) ...

>
>
> JM
>
> On Apr 14, 2013, at 7:32 PM, David Newman <[hidden email]> wrote:
>
>> DAViCal 1.1.2 installed from git with Ján Máté's 2012-11-01 and
>> 2013-04-09 patches.
>>
>> Append works now for importing large address books. Thanks Ján!
>>
>> This server now has an address book with nearly 6,000 entries, imported
>> in two parts using the append feature. The number of contacts on the
>> server agrees with the number exported from Contacts.app on OS X 10.8.3.
>> Also, I granted my server account all privileges for this resource.
>>
>> However, there are issues sharing contacts across platforms:
>>
>> 1. READ: After adding the DAViCal server as an contacts account,
>> Contacts.app on OS X sees only 714 of the ~6,000 entries.
>>
>> After trying to retrieve entries from the server, the client app
>> eventually displays an error icon (a triangle with an exclamation
>> point). Clicking this icon returns this message:
>>
>> The operation couldn’t be completed. (CoreDAVErrorDomain error 3.)
>>
>> Google is no help with that error code.
>>
>> Similarly, an iPhone and iPad both running iOS 6.1.3 both see only 1,214
>> of the ~6,000 entries. There's no error displayed.
>>
>> 2. WRITE: Adding a contact on either iOS device is possible, but the
>> contact is not added to the DAViCAL server.
>>
>> Also, the added contact disappears after navigating away from the
>> Contacts app. I don't just mean disappears from the server's group; I
>> mean the newly added contact doesn't appear under any group, even though
>> it's previously been saved.
>>
>> Similar issue with OS X; I can add a new contact, save it, and drag it
>> onto the server list,  but the number of server entries does not increase.
>>
>> Neither 1 nor 2 above produce an error on the server side*, and I've
>> verified there's nothing on the firewall between client and server that
>> is blocking communications.
>>
>> Thanks in advance for troubleshooting clues.
>>
>> dn
>>
>> *There are some entries in the error logs that pertain to the address
>> book; samples below. However, these entries are from *before* attempting
>> the read and write problems described above.
>>
>> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
>> REPORT: Query: SQ: Took: 0.344631 for SELECT collection.*,
>> calendar_item.*, caldav_data.*, addressbook_resource.*, sync_changes.*
>> FROM collection LEFT JOIN sync_changes USING(collection_id) LEFT JOIN
>> caldav_data USING (collection_id,dav_id) LEFT JOIN calendar
>> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
>> REPORT: Query: SQ: _item USING (collection_id,dav_id) LEFT JOIN
>> addressbook_resource USING (dav_id) WHERE collection.collection_id =
>> :collection_id AND sync_time >= (SELECT modification_time FROM
>> sync_tokens WHERE sync_token = :sync_token) ORDER BY collection
>> [Sun Apr 14 09:41:46 2013] [error] [client 666.1.2.3] davical: LOG:
>> REPORT: Query: SQ: .collection_id, sync_changes.dav_name,
>> sync_changes.sync_time
>>
>>
>>
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Precog is a next-generation analytics platform capable of advanced
>> analytics on semi-structured data. The platform includes APIs for building
>> apps and a phenomenal toolset for data science. Developers can use
>> our toolset for easy data analysis & visualization. Get a free account!
>> http://www2.precog.com/precogplatform/slashdotnewsletter
>> _______________________________________________
>> Davical-general mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/davical-general
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Ján Máté-2
There are 3 possible solutions:

1.) if the error occurs always at the same contact check the request/response between the client & server ... maybe there is some invalid vCard and the client fails on it (OS X and iOS vCard processing is DIFFERENT - iOS is MUCH better)
2.) if the problem is the response size (timeout), you can reduce it by enabling a compression in apache/php
3.) if the problem is the number of contacts, you need to break the addressbook into multiple smaller addressbooks


JM


On Apr 14, 2013, at 8:09 PM, David Newman <[hidden email]> wrote:

> Based on the spinning wheel icon on both client types, I'd tend to agree.
>
> Is there anything I can do on the server or client side to reduce the
> size of the request? (That is, other than the obvious solution of
> breaking one big address book into multiple smaller ones...which still
> would involve lots of responses from the server.)
>
> Thanks!
>
> dn

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

David Newman
On 4/14/13 11:19 AM, Ján Máté wrote:
> There are 3 possible solutions:
>
> 1.) if the error occurs always at the same contact check the request/response between the client & server ... maybe there is some invalid vCard and the client fails on it (OS X and iOS vCard processing is DIFFERENT - iOS is MUCH better)

Hard to see with SSL. :-)

Unencrypted, Wireshark sees Contacts.app send a RST after receiving a
lot of data. Wireshark says there's a segment missing in the next packet.

There's no sign of an error preceding the RST. This is conjecture, but
perhaps Contacts.app can buffer only so much data, and then gives up.

It would be great if there were a smoking gun, like a corrupt vCard. But
I don't see any indication of that in the packet capture, and there's
some evidence to the contrary (DAViCal imported everything OK).

Agreed that iOS is much more efficient than OS X.

> 2.) if the problem is the response size (timeout), you can reduce it by enabling a compression in apache/php

There is a small improvement with PHP zlib compression, but there are
still lots of missing vCards.

I did not see any significant improvement with Apache compression but I
can enable both Apache and php compression if you think it would help.

> 3.) if the problem is the number of contacts, you need to break the addressbook into multiple smaller addressbooks

Hmmm. I would have thought this would have worked best of all. I tried
with the letters A, B, and C and found some problems with this:

1. OS X Contacts.app displays only the A addressbook. (At least it got
all the A names.)

2. iOS shows A, B, and C addressbooks but does NOT show all vCards (740
out of 1058; A and C are complete, but B shows only 150 out of 468).

3. A new contact added in iOS gets added to the A addressbook,
regardless of last name. So eventually the A addressbook will have
retrieval problems due to size.

Given that the DAViCal server has all the addresses, I'm convinced this
is a client problem, or perhaps some problem with some vCards, or (more
likely) some config issue on my part.

Thanks for any further troubleshooting clues.

dn


>
>
> JM
>
>
> On Apr 14, 2013, at 8:09 PM, David Newman <[hidden email]> wrote:
>
>> Based on the spinning wheel icon on both client types, I'd tend to agree.
>>
>> Is there anything I can do on the server or client side to reduce the
>> size of the request? (That is, other than the obvious solution of
>> breaking one big address book into multiple smaller ones...which still
>> would involve lots of responses from the server.)
>>
>> Thanks!
>>
>> dn
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

David Newman
ps. If this is a bad vCard(s) problem, how would I tell?

thanks

clueless n00b dn


On 4/15/13 11:39 AM, David Newman wrote:

> On 4/14/13 11:19 AM, Ján Máté wrote:
>> There are 3 possible solutions:
>>
>> 1.) if the error occurs always at the same contact check the request/response between the client & server ... maybe there is some invalid vCard and the client fails on it (OS X and iOS vCard processing is DIFFERENT - iOS is MUCH better)
>
> Hard to see with SSL. :-)
>
> Unencrypted, Wireshark sees Contacts.app send a RST after receiving a
> lot of data. Wireshark says there's a segment missing in the next packet.
>
> There's no sign of an error preceding the RST. This is conjecture, but
> perhaps Contacts.app can buffer only so much data, and then gives up.
>
> It would be great if there were a smoking gun, like a corrupt vCard. But
> I don't see any indication of that in the packet capture, and there's
> some evidence to the contrary (DAViCal imported everything OK).
>
> Agreed that iOS is much more efficient than OS X.
>
>> 2.) if the problem is the response size (timeout), you can reduce it by enabling a compression in apache/php
>
> There is a small improvement with PHP zlib compression, but there are
> still lots of missing vCards.
>
> I did not see any significant improvement with Apache compression but I
> can enable both Apache and php compression if you think it would help.
>
>> 3.) if the problem is the number of contacts, you need to break the addressbook into multiple smaller addressbooks
>
> Hmmm. I would have thought this would have worked best of all. I tried
> with the letters A, B, and C and found some problems with this:
>
> 1. OS X Contacts.app displays only the A addressbook. (At least it got
> all the A names.)
>
> 2. iOS shows A, B, and C addressbooks but does NOT show all vCards (740
> out of 1058; A and C are complete, but B shows only 150 out of 468).
>
> 3. A new contact added in iOS gets added to the A addressbook,
> regardless of last name. So eventually the A addressbook will have
> retrieval problems due to size.
>
> Given that the DAViCal server has all the addresses, I'm convinced this
> is a client problem, or perhaps some problem with some vCards, or (more
> likely) some config issue on my part.
>
> Thanks for any further troubleshooting clues.
>
> dn
>
>
>>
>>
>> JM
>>
>>
>> On Apr 14, 2013, at 8:09 PM, David Newman <[hidden email]> wrote:
>>
>>> Based on the spinning wheel icon on both client types, I'd tend to agree.
>>>
>>> Is there anything I can do on the server or client side to reduce the
>>> size of the request? (That is, other than the obvious solution of
>>> breaking one big address book into multiple smaller ones...which still
>>> would involve lots of responses from the server.)
>>>
>>> Thanks!
>>>
>>> dn
>>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Vincent Van Houtte
Hi,

On 16-04-2013 02:04, David Newman wrote:
> ps. If this is a bad vCard(s) problem, how would I tell?

I haven't tried it, but the article seems to indicate this guy knows
what he's doing :)

http://l0b0.wordpress.com/2009/12/25/vcard-parser-and-validator/

HTH,
Vincent
--
Mvg,
Vincent Van Houtte
Advocaat
Advocatenkantoor Suy, Van Baeveghem & Van Houtte
Brusselsestraat 108
9200 Dendermonde
T 052 52 06 05
F 052 52 06 46
W http://synergylaw.be

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

David Newman
On 4/15/13 11:04 PM, Vincent Van Houtte wrote:
> Hi,
>
> On 16-04-2013 02:04, David Newman wrote:
>> ps. If this is a bad vCard(s) problem, how would I tell?
>
> I haven't tried it, but the article seems to indicate this guy knows
> what he's doing :)
>
> http://l0b0.wordpress.com/2009/12/25/vcard-parser-and-validator/

Thanks! That's a great resource.

I made a little progress with this, but am still seeing parser errors on
both OS X and iOS clients.

The parser script (which runs only after adding an extra line to the end
of a vCard file) complains about long lines in multiple vCard entries.
RFC 6350 does not forbid long lines, but says lines SHOULD be no longer
than 75 characters.

Just breaking an offending line into two with a CRLF did not help, even
though RFC 6350 says such formatting is OK. However, adding a backslash
at the end of a line DID allow two previously "bad" vCards to be
imported into iOS.

I wrote a little perl script to truncate all long lines at 70
characters, add a backslash, and print out the rest. That script is at
the end of this message.

Even after truncation, iOS and OS X still complain about some vCards.
The parser script says the vCard file is OK aside from warnings about
some "short folded lines" (the ones created by truncation).

Someone suggested checking the console logs via Console.app and the
iPhone Configuration Utility. The ICU just says there are parser errors,
but offers no details. Console.app is a little better; it says it
ignored a particular vCard because "ABPerson create failed."

Looking at the vCard file, I don't see anything unusual about that
person's ABPerson record:

X-ABUID:AB1810FF-59A7-419B-831D-B01326BFF54A:ABPerson

I'm pretty close to concluding Apple's Mac and iOS addressbook clients
don't play nice with the vCard standard, and thus aren't good to use
with DAViCal. I don't want to give up on this, but I'm pretty close.

Alternative suggestions gladly accepted.

Thanks!

dn




perl truncation script:

#!/usr/bin/perl

use strict;
use warnings;

# truncate.pl -- truncate vcf files to 70 chars

my $file = "B468.vcf";

my $lengthOK = 70;

open FILE, $file or die $!;
my @data = <FILE>;
close(FILE);

foreach my $line (@data) {
        # lose the Windoze CRLFs
        $line =~ s/\015?\012?$//;
       
        # measure length
        my $length = length $line;
       
        # just print lines of 70 chars or less
        if ($length <= $lengthOK) {
                printf "$line\n";
        }

        # truncate lines > 70 chars, then print
        else {
                $line = join("\\\n", unpack("(A70)*", $line));
                printf "$line\n";
        }
}

nb. I converted the output to use CRLFs before importing into DAViCal.



------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Vincent Van Houtte
Hi,

> I'm pretty close to concluding Apple's Mac and iOS addressbook
> clients
> don't play nice with the vCard standard, and thus aren't good to use
> with DAViCal. I don't want to give up on this, but I'm pretty close.

The link I sent you earlier was not the resource I was hoping to find
back. A while ago I found an article by someone who was having problems
with this (not DAViCal related though) and the conclusion of his
extensive research was that all major players on the market have their
own interpretation of the vCard-'standard'.

His proposal was to make a new (v5 iirc) standard or recommendation,
but a strict one. Until then, there is little or no interoperability
between Apple iOS, Apple OSX, Google Mail Contacts, Yahoo etc.

HTH,
Vincent
--
Mvg,
Vincent Van Houtte
Advocaat
Advocatenkantoor Suy, Van Baeveghem & Van Houtte
Brusselsestraat 108
9200 Dendermonde
T 052 52 06 05
F 052 52 06 46
W http://synergylaw.be

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Ján Máté-2
In reply to this post by David Newman
Hi David,

try to check the difference between your vCards and identical vCards created by CardDavMATE (ignore the differences in PRODID, UID and REV values).

One of the main goals of my client is Apple compatibility and we (~50 people) never noticed any problem with vCards created by CardDavMATE in iOS.


JM


On Apr 16, 2013, at 11:45 PM, David Newman <[hidden email]> wrote:

> On 4/15/13 11:04 PM, Vincent Van Houtte wrote:
>> Hi,
>>
>> On 16-04-2013 02:04, David Newman wrote:
>>> ps. If this is a bad vCard(s) problem, how would I tell?
>>
>> I haven't tried it, but the article seems to indicate this guy knows
>> what he's doing :)
>>
>> http://l0b0.wordpress.com/2009/12/25/vcard-parser-and-validator/
>
> Thanks! That's a great resource.
>
> I made a little progress with this, but am still seeing parser errors on
> both OS X and iOS clients.
>
> The parser script (which runs only after adding an extra line to the end
> of a vCard file) complains about long lines in multiple vCard entries.
> RFC 6350 does not forbid long lines, but says lines SHOULD be no longer
> than 75 characters.
>
> Just breaking an offending line into two with a CRLF did not help, even
> though RFC 6350 says such formatting is OK. However, adding a backslash
> at the end of a line DID allow two previously "bad" vCards to be
> imported into iOS.
>
> I wrote a little perl script to truncate all long lines at 70
> characters, add a backslash, and print out the rest. That script is at
> the end of this message.
>
> Even after truncation, iOS and OS X still complain about some vCards.
> The parser script says the vCard file is OK aside from warnings about
> some "short folded lines" (the ones created by truncation).
>
> Someone suggested checking the console logs via Console.app and the
> iPhone Configuration Utility. The ICU just says there are parser errors,
> but offers no details. Console.app is a little better; it says it
> ignored a particular vCard because "ABPerson create failed."
>
> Looking at the vCard file, I don't see anything unusual about that
> person's ABPerson record:
>
> X-ABUID:AB1810FF-59A7-419B-831D-B01326BFF54A:ABPerson
>
> I'm pretty close to concluding Apple's Mac and iOS addressbook clients
> don't play nice with the vCard standard, and thus aren't good to use
> with DAViCal. I don't want to give up on this, but I'm pretty close.
>
> Alternative suggestions gladly accepted.
>
> Thanks!
>
> dn
>
>
>
>
> perl truncation script:
>
> #!/usr/bin/perl
>
> use strict;
> use warnings;
>
> # truncate.pl -- truncate vcf files to 70 chars
>
> my $file = "B468.vcf";
>
> my $lengthOK = 70;
>
> open FILE, $file or die $!;
> my @data = <FILE>;
> close(FILE);
>
> foreach my $line (@data) {
> # lose the Windoze CRLFs
> $line =~ s/\015?\012?$//;
>
> # measure length
> my $length = length $line;
>
> # just print lines of 70 chars or less
> if ($length <= $lengthOK) {
> printf "$line\n";
> }
>
> # truncate lines > 70 chars, then print
> else {
> $line = join("\\\n", unpack("(A70)*", $line));
> printf "$line\n";
> }
> }
>
> nb. I converted the output to use CRLFs before importing into DAViCal.
>
>
>
> ------------------------------------------------------------------------------
> Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for building
> apps and a phenomenal toolset for data science. Developers can use
> our toolset for easy data analysis & visualization. Get a free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter
> _______________________________________________
> Davical-general mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general

smime.p7s (6K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Narcis Garcia - GiLUG
In reply to this post by David Newman
The user-agent cannot be detected by Davical? If not, should be deduced
on transferred content.

As an example, websites often adapt some HTML formats to user-agent,
because webbrowsers also have their own interpretation of the HTML
standards.


Al 17/04/13 09:28, En/na Vincent Van Houtte ha escrit:

> Hi,
>
>> I'm pretty close to concluding Apple's Mac and iOS addressbook
>> clients
>> don't play nice with the vCard standard, and thus aren't good to use
>> with DAViCal. I don't want to give up on this, but I'm pretty close.
>
> The link I sent you earlier was not the resource I was hoping to find
> back. A while ago I found an article by someone who was having problems
> with this (not DAViCal related though) and the conclusion of his
> extensive research was that all major players on the market have their
> own interpretation of the vCard-'standard'.
>
> His proposal was to make a new (v5 iirc) standard or recommendation,
> but a strict one. Until then, there is little or no interoperability
> between Apple iOS, Apple OSX, Google Mail Contacts, Yahoo etc.
>
> HTH,
> Vincent
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

Vincent Van Houtte
In reply to this post by David Newman
Hi,

> The user-agent cannot be detected by Davical? If not, should be
> deduced
> on transferred content.
>
> As an example, websites often adapt some HTML formats to user-agent,
> because webbrowsers also have their own interpretation of the HTML
> standards.

Yes, I guess that would be a good workaround. It would require:
* user-agent checking
* incorporating all vCard standards, which might prove to be
non-trivial

One minor problem: there is noone actively developing DAViCal...

--
Mvg,
Vincent Van Houtte
Advocaat
Advocatenkantoor Suy, Van Baeveghem & Van Houtte
Brusselsestraat 108
9200 Dendermonde
T 052 52 06 05
F 052 52 06 46
W http://synergylaw.be

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

mgia
In reply to this post by David Newman
Hi.
On 04/16/2013 02:04 AM, David Newman wrote:
> ps. If this is a bad vCard(s) problem, how would I tell?

Mac OSX has an app for debugging, konsole.
In my expierence any vCard errors should be displayed there.

-
mgia



------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: CardDAV read and write issues from Apple clients

David Newman
In reply to this post by Ján Máté-2
On 4/17/13 12:35 AM, Ján Máté wrote:
> Hi David,
>
> try to check the difference between your vCards and identical vCards created by CardDavMATE (ignore the differences in PRODID, UID and REV values).
>
> One of the main goals of my client is Apple compatibility and we (~50 people) never noticed any problem with vCards created by CardDavMATE in iOS.

This might a problem with PostgreSQL.

I imported ~3500 contacts and tried to read them in CardDavMATE. Unlike
the Apple clients, which displayed at least some clients, CardDavMATE
displayed nothing.

The Apache error log complains about a possible slow query (partially
obfuscated log entries below).

This has come up before on this list:

http://lists.davical.org/pipermail/davical-users/2010q1/000412.html

Andrew McMillan suggests restarting the database and vacuuming. But I
don't understand his ANALYZE VERBOSE suggestion, since it won't take
SELECT as an argument. This is on version 8.4.17.

Also, running just the first couple of hrefs in the query returns an error:

davical=# SELECT calendar_item.*, addressbook_resource.*, caldav_data.*
FROM caldav_data LEFT JOIN calendar_item USING(dav_id, user_no,
dav_name, collection_id) LEFT JOIN addressbook_resource USING(dav_id)
LEFT JOIN collection USING(collection_id) WHERE
caldav_data.collection_id = 53751 AND caldav_data.dav_name IN (:href0,
:href1, :href2);
ERROR:  syntax error at or near ":"
LINE 1: ...llection_id = 53751 AND caldav_data.dav_name IN ( :href0, :h...

I'm relatively new to PostgreSQL. How to resolve slow queries?

dn





[Fri Apr 19 11:54:03 2013] [error] [client 205.147.16.129] davical: LOG:
REPORT: Query: Possible slow query: SQ in
'/usr/local/www/davical/inc/caldav-REPORT-multiget.php' on line 108,
referer: https://some.example.com:8443/carddavmate/index.html
[Fri Apr 19 11:54:03 2013] [error] [client 205.147.16.129] davical: LOG:
REPORT: Query: SQ: Took: 0.449987 for SELECT calendar_item.*,
addressbook_resource.*, caldav_data.* FROM caldav_data LEFT JOIN
calendar_item USING(dav_id, user_no, dav_name, collection_id) LEFT JOIN
addressbook_resource USING(dav_id) LEFT JOIN collection USIN, referer:
https://some.example.com:8443/carddavmate/index.html
[Fri Apr 19 11:54:03 2013] [error] [client 205.147.16.129] davical: LOG:
REPORT: Query: SQ: G(collection_id) WHERE caldav_data.collection_id =
53751 AND caldav_data.dav_name IN ( :href0, :href1, :href2, :href3,
:href4, :href5, :href6, :href7, :href8, :href9, :href10, :href11,
:href12, :href13, :href14, :href15, :href16, :href17, :, referer:
https://some.example.com:8443/carddavmate/index.html

This is followed by thousands more hrefs, all part of the query.

>
>
> JM
>
>
> On Apr 16, 2013, at 11:45 PM, David Newman <[hidden email]> wrote:
>
>> On 4/15/13 11:04 PM, Vincent Van Houtte wrote:
>>> Hi,
>>>
>>> On 16-04-2013 02:04, David Newman wrote:
>>>> ps. If this is a bad vCard(s) problem, how would I tell?
>>>
>>> I haven't tried it, but the article seems to indicate this guy knows
>>> what he's doing :)
>>>
>>> http://l0b0.wordpress.com/2009/12/25/vcard-parser-and-validator/
>>
>> Thanks! That's a great resource.
>>
>> I made a little progress with this, but am still seeing parser errors on
>> both OS X and iOS clients.
>>
>> The parser script (which runs only after adding an extra line to the end
>> of a vCard file) complains about long lines in multiple vCard entries.
>> RFC 6350 does not forbid long lines, but says lines SHOULD be no longer
>> than 75 characters.
>>
>> Just breaking an offending line into two with a CRLF did not help, even
>> though RFC 6350 says such formatting is OK. However, adding a backslash
>> at the end of a line DID allow two previously "bad" vCards to be
>> imported into iOS.
>>
>> I wrote a little perl script to truncate all long lines at 70
>> characters, add a backslash, and print out the rest. That script is at
>> the end of this message.
>>
>> Even after truncation, iOS and OS X still complain about some vCards.
>> The parser script says the vCard file is OK aside from warnings about
>> some "short folded lines" (the ones created by truncation).
>>
>> Someone suggested checking the console logs via Console.app and the
>> iPhone Configuration Utility. The ICU just says there are parser errors,
>> but offers no details. Console.app is a little better; it says it
>> ignored a particular vCard because "ABPerson create failed."
>>
>> Looking at the vCard file, I don't see anything unusual about that
>> person's ABPerson record:
>>
>> X-ABUID:AB1810FF-59A7-419B-831D-B01326BFF54A:ABPerson
>>
>> I'm pretty close to concluding Apple's Mac and iOS addressbook clients
>> don't play nice with the vCard standard, and thus aren't good to use
>> with DAViCal. I don't want to give up on this, but I'm pretty close.
>>
>> Alternative suggestions gladly accepted.
>>
>> Thanks!
>>
>> dn
>>
>>
>>
>>
>> perl truncation script:
>>
>> #!/usr/bin/perl
>>
>> use strict;
>> use warnings;
>>
>> # truncate.pl -- truncate vcf files to 70 chars
>>
>> my $file = "B468.vcf";
>>
>> my $lengthOK = 70;
>>
>> open FILE, $file or die $!;
>> my @data = <FILE>;
>> close(FILE);
>>
>> foreach my $line (@data) {
>> # lose the Windoze CRLFs
>> $line =~ s/\015?\012?$//;
>>
>> # measure length
>> my $length = length $line;
>>
>> # just print lines of 70 chars or less
>> if ($length <= $lengthOK) {
>> printf "$line\n";
>> }
>>
>> # truncate lines > 70 chars, then print
>> else {
>> $line = join("\\\n", unpack("(A70)*", $line));
>> printf "$line\n";
>> }
>> }
>>
>> nb. I converted the output to use CRLFs before importing into DAViCal.
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Precog is a next-generation analytics platform capable of advanced
>> analytics on semi-structured data. The platform includes APIs for building
>> apps and a phenomenal toolset for data science. Developers can use
>> our toolset for easy data analysis & visualization. Get a free account!
>> http://www2.precog.com/precogplatform/slashdotnewsletter
>> _______________________________________________
>> Davical-general mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/davical-general
>

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general
Reply | Threaded
Open this post in threaded view
|

Re: SOLVED: CardDAV read and write issues from Apple clients

David Newman
In reply to this post by mgia
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 4/17/13 5:58 AM, mgia wrote:
> Hi. On 04/16/2013 02:04 AM, David Newman wrote:
>> ps. If this is a bad vCard(s) problem, how would I tell?
>
> Mac OSX has an app for debugging, konsole. In my expierence any
> vCard errors should be displayed there.

Thanks. The root problem turned out to be garbage characters in some
vCards exported from Contacts.app.

OS X Console.app isn't very helpful -- it just says there's a parsing
error but doesn't say what the error is. At least it provides the
vCard that it doesn't like. The iPhone Configuration Utility is not as
good; it too says there's a parsing error, but doesn't provide a vCard
name.

With Ján Máte's help, I did some Javascript debugging using
CardDAVMate and Firefox's error console. That provided exact line and
column references for each bad character.

I then used a hex editor to see what was at each location -- escape
characters, vertical tabs, backspaces, and a few other characters that
couldn't be read. Deleting each bad character allowed CardDAVMate to
load a bit more of the vCard list before throwing the next error.

This was a bit painstaking, as I'd have to delete, edit, and then
re-create the contacts database for each offending character. There
were about 30 bad characters in a ~6000-vCard database, mostly stuff
that had been pasted into the Notes field from who knows where.

With all the offending characters gone, every client now displays all
contacts -- CardDAVMate, OS X Contacts.app, and iOS clients on iPhones
and iPads.

Thanks all for your help in getting this issue resolved.

dn




>
> - mgia
>
>
>
> ------------------------------------------------------------------------------
>
>
Precog is a next-generation analytics platform capable of advanced
> analytics on semi-structured data. The platform includes APIs for
> building apps and a phenomenal toolset for data science. Developers
> can use our toolset for easy data analysis & visualization. Get a
> free account!
> http://www2.precog.com/precogplatform/slashdotnewsletter 
> _______________________________________________ Davical-general
> mailing list [hidden email]
> https://lists.sourceforge.net/lists/listinfo/davical-general
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.18 (Darwin)
Comment: GPGTools - http://gpgtools.org
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iEYEARECAAYFAlF0I2UACgkQyPxGVjntI4JR+gCdG9VxQAsa4wmp+2wFYxy8CTaP
uWUAmQFIlIIjoJP5wb8y3gPR8KrOwFuJ
=E0Oa
-----END PGP SIGNATURE-----

------------------------------------------------------------------------------
Precog is a next-generation analytics platform capable of advanced
analytics on semi-structured data. The platform includes APIs for building
apps and a phenomenal toolset for data science. Developers can use
our toolset for easy data analysis & visualization. Get a free account!
http://www2.precog.com/precogplatform/slashdotnewsletter
_______________________________________________
Davical-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/davical-general